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ABSTRACT 



In 1996, Quest staff began working with teams from school 
communities in three West Virginia county school districts to invigorate 
efforts for continuous school improvement. This first learning community 
consisted of students, teachers, administrators, parents, and community 
members, who ultimately wrote individual school visions and improvement plans 
and co-authored a supplemental guide for state-mandated school improvement 
plans. Tinder Elementary became involved in Quest in 1997 when four staff 
members and one parent attended the first elementary network rally. The 
school also involved Quest staff in two co- ventures in learning, during which 
Quest staff interviewed teachers and students, observed 24 classrooms, and 
planned with parents the piloting of a series of sessions on Successful, 
Motivated, Autonomous, Responsible, and Thoughtful (SMART) learners. Although 
the school was highly involved in the project in many respects, there were 
some limitations to their participation. The project also had little impact 
on student achievement, though positive trends in grades 3 through 5 were 
apparent, and a few statistically significant improvements were located. 
Student Test of Cognitive Skills/2 total, nonverbal, and verbal scores were 
consistently higher in 1998 and 1999 than in 1996, when Tinder began its 
involvement in Quest. Appendixes contain the Quest brochure and framework; 
interview protocols for individual, group team members, and student focus 
groups; and a questionnaire, checklist, and instrument. (Contains 44 
references and 17 tables.) (DFR) 
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INTRODUCTION 



AEL’s Quest Project 

As part of its contract to develop a framework for continuous school improvement in its four- 
state region, AEL staff designed the Quest project (see Appendix A). Based upon principles of 
inquiry, collaboration, and action research, Quest proposes to support and investigate ongoing school 
improvement efforts through twice-yearly conferences (which staff renamed rallies), summer 
symposia, a Scholars program, visits to participating schools, communication via listserv and 
mailings, and the creation of a Quest network of schools. 

The project draws from literature on school change suggesting that subjectivity and personal 
growth are essential to the change process (Fullan, 1991). Yet because individual development takes 
place within a variety of social contexts, including school communities, staff designed the Quest 
network with attention to the ways shared vision, goals, and sense of community support ongoing 
school improvement (Barth, 1990; Hord, Rutherford, Huling-Austin, & Hall, 1987; Postman, 1995; 
Sergiovanni, 1994). Similarly, school culture may impede or enhance significantly the viability of 
school improvement work (Richardson, 1996; Ryan, 1995). If a school community shares certain 
norms, such as self-evaluation, curiosity, proactivity, and high performance expectations, reform 
efforts are hypothesized to fare better than those in school cultures that do not possess such norms. 
Other research suggests that school administrators must assume a collaborative role in decision- 
making if reform efforts are to succeed (van der Bogert, 1998), and that instructional and curricular 
goals must be informed by a diverse contingent of school stakeholders, including parents, students, 
and community members (Barth, 1990; Sergiovanni, 1994). 

Quest staff were also attuned to literature suggesting that honoring the purpose of education 
enhances school change. Assessment strategies, for instance, ought to serve multiple ends, not the 
least of which is to provide information for ongoing teaching and learning (Wiggins, 1993). 
Moreover, education generally and reform endeavors specifically need to nurture a host of attributes 
enabling students to make use of their education to lead thoughtful lives (Perkins, 1995; Postman, 
1995). 



In sum, Quest staff sought to create a network of schools committed to continuous 
improvement, collaboration, and inquiry. Participants would engage in, reflect upon, and assess the 
reform endeavors their schools undertook with the support of Quest. 

Quest Activities 

In the summer of 1996, Quest staff at AEL began working with teams from school 
communities in three West Virginia county school districts to invigorate efforts for continuous 
school improvement, using a variety of techniques for gathering input from all those with a stake in 
their local schools (Howley-Rowe, 1998g). This first learning community, called Leadership to 
Unify School Improvement Efforts (LUSIE), consisted of school teams including students, teachers, 
administrators, parents, and community members. Ultimately, this group wrote individual school 
visions and improvement plans and co-authored (with AEL) Creating Energy for School 
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Improvement (1997), a supplemental guide for those poised to write their own state-mandated school 
improvement plans. 

Quest staff also were committed to creating learning communities devoted to exploring 
continuous school improvement across the AEL region of Kentucky, Tennessee, Virginia, and West 
Virginia. Hence, staff scheduled a pilot Inquiry Into Improvement conference in April 1997 for 
selected region high schools. Schools were selected in several ways. Some schools were 
recommended for the Quest experience by central office staff or school administrators. Other schools 
were asked to join Quest because they had participated in previous AEL programs. Still other 
schools were invited because Quest staff believed they were primed for the kind of collaborative 
inquiries into school improvement that Quest was designed to provide. 

In October 1997, in Roanoke, Virginia, another conference was held for designated high 
schools in the AEL region, this time with an explicit emphasis on forming and nurturing a network 
of schools (Howley-Rowe, 1998c). A similar conference was held in Nashville, Tennessee, for 
designated regional elementary schools in November 1997 (Howley-Rowe, 1998a). In order to 
facilitate the development of a Quest school network and to continue to help invigorate continuous 
school improvement efforts within network schools, staff planned a sequence of events in 1998 
following these initial conferences. Dissatisfied with the conventional and prescriptive connotation 
of the word conference, Quest staff chose to call these network meetings rallies. Thus, all events 
previously called conferences are now termed rallies. 

The high school network met a second time on February 8-10, 1998, at the Pipestem State 
Park Resort in West Virginia (Howley-Rowe, 1998d), following which the elementary school 
network participated in a rally on February 22-24, 1998, in Lexington, Kentucky (Howley-Rowe, 
1998b). During the summer, 11 network members participated in the Quest Scholars Program, 
meeting at a colloquium in Charleston, West Virginia, on July 16-18, 1998, to collaborate with 
project staff in ongoing efforts to conceptualize, design, and research Quest (Howley-Rowe, 1 998e). 
Finally, in August, network members and other educators in AEL’s region participated in a 
symposium on assessment of student work (Howley-Rowe, 1998f). 

From the high school network rally in October 1 997 to the August 1998 summer symposium, 
Quest staff hosted six network events. The Quest network contained an essentially stable 
membership, although there were differences in the number of school teams that attended each event 
and in the frequency that school teams attended gatherings. Project staff recently investigated this 
phenomenon, finding that administrative support for participation in the network was the factor 
reported to be most important to schools’ initial and sustained involvement in Quest (Howley-Rowe, 
1999c). • 



Beginning their second year of network activity, Quest staff invited the elementary and high 
school networks to attend a rally together on November 2-3, 1 998, at the Glade Springs Resort, near 
Daniels, West Virginia (Howley-Rowe, 1999a). Approximately half of the Quest Scholars met on 
November 1, 1998, to plan with project staff several rally activities. Scholars from the high school 
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network met for three hours on February 14, 1999, prior to a high school network rally held on 
February 15-16 in Roanoke, Virginia (Howley-Rowe, 1999c). A similar rally was held for 
elementary network members on February 22-23, 1999, in Lexington, Kentucky (Howley-Rowe, 
1999b). 



A second Scholars colloquium was convened from July 12-15, 1999, at Mountain Lake 
Resort, Virginia (Howley-Rowe, 1 999c). The primary purpose of this colloquium was for Quest staff 
and Scholars to collaborate in evaluating and writing about the project, ultimately contributing 
written pieces to a publication about the Quest network. In addition, a second summer symposium 
was convened in Gatlinburg, Tennessee, July 26-27, 1999 (Parrish & Howley-Rowe, 2000). 

The third year of Quest events began with two rallies and a Scholars meeting in November 
1999 in Bristol, Virginia. A rally for elementary schools was conducted from November 11-12, 
1999. Scholars met to discuss writing and several Quest instruments November 13-14. And a high 
school rally was held November 15-16. Network high schools met again from February 14-15, 2000, 
in Roanoke, Virginia. Elementary schools participated in a rally from February 17-18, 2000, in 
Lexington, Kentucky. Evaluation of these events was not conducted as staff turned their efforts to 
summative evaluation of the project; Quest and the 1996-2000 REL contract funding the project 
would come to an end in November 2000. 

Summative Evaluation of Quest 

Quest staff delineated several evaluation questions they hoped summative evaluation would 
address. These questions were categorized in terms of inputs and outputs, or independent and 
dependent variables. In other words, staff wanted to understand the relationships between issues such 
as the extent of involvement in Quest and school-specific improvement efforts inspired by Quest, 
as well as the extent to which professional learning community was enhanced or to which 
participating schools approximated the Quest framework of continuous improvement. More 
succinctly. Quest staff hoped to learn from summative evaluation what impact participation in Quest 
had upon schools, individuals within them, and upon the network as a whole. 

Summative evaluation questions and the instruments or methods used to answer them are 
listed in Table 1. 

Thus, summative evaluation of Quest is intended to answer the questions delineated in Table 
1 formulated by Quest staff and the evaluator. Summative evaluation will describe the impact Quest 
had upon schools and their school communities, providing some evidence of the effectiveness of the 
project. 
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Table 1 

Summative Evaluation Questions and Instruments/Methods 
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Independent variables/inputs 


Instruments/Methods 


To what extent do Quest team members think their 
schools have enacted Quest components? 


Innovation Configuration Checklist 


What specific Quest related activities have schools 
participated in? 


History of involvement 


What have schools undertaken as a result of Quest? 


Faculty focus group 

Quest team member interviews 


Dependent variables/outcomes 


Instruments/Methods 


Has Quest enhanced professional learning 
community in network schools? 


School Staff as Professional Learning Community 
instrument pre- and post-test 


To what extent do members of the Quest team think 
their school approximates the Quest framework, and 
to what degree is this attributable to Quest? 


Reflective Assessment instrument 


What have been the changes in student achievement 
during Quest participation? 


Achievement data 
School report card 


What has been the impact of Quest on individuals, 
schools, and of what value has the network been? 


Quest team member interviews 


What have schools undertaken as a result of Quest, 
and what have been the results? 


Faculty focus group 

Quest team member interviews 


What have been the results of school projects 
undertaken due to Quest? 


School data about results of school projects 


What do network participants report has happened 
at their schools due to Quest? In what other ways 
has Quest been effective? 


School stories 



The primary audience for summative evaluation of the project is Quest staff at AEL. It is 
intended to offer project staff a summative perspective on the impact of Quest in four disparate 
schools. Other audiences include representatives of AEL’s funding source, the U.S. Department of 
Education’s Office of Educational Research and Improvement (OERI), and policymakers, school 
administrators, teachers, education researchers, and others interested in strategies to support 
continuous school improvement. 
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METHODS 

A case study approach was taken for summative evaluation of the Quest project. Given that 
Quest staff were most interested in understanding the impact of the project on various levels, from 
the individual to the school to the network, the case study method seemed appropriate. In addition, 
project staff were committed to understanding project impact from the perspectives of various 
participants in the network, including students, teachers, parents, and administrators. Case studies 
involve in-depth “multi-perspectival analyses” (Tellis, 1997) of single systems or phenomena; they 
rely on clearly delineated boundaries rather than on sampling (Stake, 1995). The focus, depth, and 
ability to account for multiple viewpoints associated with the case study approach led Quest staff to 
consider using such a method. 

Moreover, formative evaluation had revealed the high level of satisfaction participants had 
with Quest and the great extent to which the project met its goals at each event (Howley-Rowe, 
1999a-c, 1998a-f). Exploratory research also indicated various reasons some schools were more 
involved in the network than others (Howley-Rowe, 1999d). These sources of information convinced 
project staff that Quest had made some impact on those involved. Quest staff were, therefore, more 
interested in summative evaluation that elucidated in what ways Quest had been of value to schools 
and individuals in the project than in evaluation focusing solely on quantitative outcome measures. 

Hence, summative evaluation of the Quest project includes case studies of four network 
schools. The schools were selected for their high level of involvement in Quest, varied 
interpretations and uses of the project, and diverse locations and demographic constitutions. 

For example, while one school used Quest to support parent involvement programs, another 
discovered a variety of inquiry techniques to improve student writing. One small elementary school 
is located in a rural, impoverished Appalachian area, whereas a very large high school is in a 
relatively wealthy suburb of the nation’s capital. All four schools, nonetheless, found Quest flexible 
enough to accommodate their very different goals for improvement and structured enough to provide 
constructive strategies supporting change. 

A strength of case studies is their reliance on triangulation of data to provide a more 
comprehensive description of the objects of study than might be rendered by use of a single research 
method. Using several data sources in order to corroborate theses is what Brewer and Hunter (1989) 
call “multimethod research.” This approach posits that the strengths of each method will compensate 
for the weaknesses in others, ultimately providing a more complete account of that being studied. 

On the other hand, the case study approach has been criticized for its “dependence on a single 
case [which] renders it incapable of providing a generalizing conclusion” (Tellis, 1997, p. 3). 
Although generalization to populations is certainly compromised by the case study method, 
generalization to theory is not as problematic if case studies are conducted with sufficient rigor and 
transparency. Hence, conclusions, generated by case studies can be used to generalize by synecdoche 
as “a claim that the essential features of the larger social unit are reproduced in microcosm within 
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the smaller social unit, and that by studying them in micro we might make inferences about the 
macrostructure of which they are a part” (Brewer & Hunter, 1989, p. 123). 

Both qualitative and quantitative methods contributed to this evaluation component of the 
Quest project. During project events, the evaluator engaged in participant observation (Becker & 
Geer, 1957; Emerson, 1983; Glazer, 1972; Hammersley & Atkinson, 1983; Miles & Huberman, 
1994), a method highly suited “for studying processes, relationships among people and events, the 
organization of people and events, continuities over time, and patterns” (Jorgensen, 1989, p. 12). 
Furthermore, consistent with the Quest paradigm, participant observation involves “a flexible, open- 
ended, opportunistic process and logic of inquiry through which what is studied constantly is subject 
to redefinition based on field experience and observation” (Jorgensen, 1 989, p. 23). This method “is 
a commitment to adopt the perspective of those studied by sharing in their . . . experiences” (Denzin, 
1989, p. 156), thereby enabling researchers to evaluate how an event or process appears and feels 
to participants. Finally, participant observation places the evaluator squarely in the field, rather than 
in the office or on the phone, allowing for the collection of richer, more directly acquired data 
(Patton, 1980). 

Denzin (1989) describes four variations in participant observation strategies: the complete 
participant, the participant as observer, the observer as participant, and the complete observer (pp. 
162-65). The evaluator played a role more akin to the participant as observer, participating in 
ongoing project activities as appropriate but not concealing data collection. 

In order to corroborate the theses generated by participant observation, the evaluator also 
performed other data collection and analysis activities. The evaluator and a trained Quest consultant 
conducted a site visit to Tinder Elementary School to gather data on May 4-5, 2000. Semi-structured 
individual interviews were conducted with six Tinder Quest team members using a predesigned 
protocol. In addition, semi-structured group interviews were conducted with approximately 12 
members of the school faculty who had been minimally or not at all involved in Quest events. A 
semi-structured group interview with 12 students was conducted as well. Finally, six Quest team 
members completed the Reflective Assessment questionnaire. 

Pre- and post-test scores on the School Professional Staff as Learning Community were 
analyzed to discern if case study schools had become more like professional learning communities 
over the course of their participation in Quest. This instrument was first administered to all network 
schools in December 1 997, and again in November 1 999, as the project drew to a close. The surveys 
were sent to a contact person at each network school who distributed the instruments to faculty, then 
collected and returned completed surveys to Quest staff. 

Another instrument completed by Quest participants at the close of the project was an 
Innovation Configuration Checklist detailing the essential components of Quest as well as variations 
thereof. All network participants in attendance at the February 2000 rallies were asked to complete 
the Checklist. 
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Other data sources included academic aptitude data from the state-mandated standardized 
Test of Cognitive Skills and data gathered during Tinder participation in Quest sponsored events, 
including a technical assistance visit and project events. 

Identical instruments and individual and group interview protocols were used across the four 
case study sites to allow for comparative analyses, should staff consider such comparisons useful. 
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TINDER ELEMENTARY SCHOOL 
Context 

Tinder Elementary 1 is located in a suburban neighborhood of a mid-size Kentucky city. The 
school building opened in the fall of 1 990 and currently serves approximately 700 students in grades 
kindergarten through sixth. Located on a 30-acre campus, the school includes 1 5 acres with walking 
trails, fitness stations, a playground, wild flower gardens, and a show garden of iris maintained by 
local iris gardeners. 

Eighty-six percent of the student population at Tinder is White, 7% African American, 4% 
of Asian descent, and 2% Latino. The school does not qualify for Title I funds; 1 6.2% of the students 
qualify for free or reduced price lunches. 

The school is served by 54 staff, 43 of whom are certified. A principal, assistant principal, 
41 teachers, and 1 1 other instructional staff comprise this group. Classroom pupil-teacher ratios 
range from 1 :22 to 1 :28, depending on the class. 

Participation in Quest 

Tinder became involved in Quest inNovember 1 997 when four staff members and one parent 
attended the first elementary network rally. The school sent a leadership team to each following rally, 
with membership of five to seven. One team member participated in the 1 998 Scholars colloquium, 
and two in the 1999 Scholars program. Ten Tinder school community members attended the July 
1 999 summer symposium. 

The school also involved Quest staff in two co-ventures in learning, during which Quest staff 
interviewed teachers and students, observed 24 classrooms, planned with parents the piloting of a 
series of sessions on SMART 2 learners and SMART parenting, and provided technical assistance 
with the implementation of student-led conferences. 

In October 1 999, Tinder staff collaborated with AEL on an action research project concerning 
math instruction. In addition, the school adopted the SMART learner component of the Quest 
framework as a theme, using the phrase in hallway displays and on tee-shirts. 

At the November 1 998 elementary and high school rally at Glade Springs Resort, Tinder 
principal Dr. Dottie Baldwin shared the school’s use of the SMART learner theme: 



'All school and personal names used in this report are pseudonyms in order to protect the 
anonymity and confidentiality of participants. 

2 Developed by Quest staff, SMART is an acronym for Successful, Motivated, 
Autonomous, Responsible, and Thoughtful. 
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“At 2:07, the facilitators introduced another form of sharing within the network. Dottie 
[Baldwin], a principal from Tinder Elementary, shared with participants a slide show of her 
school, telling the story of its development and current undertakings. Notably, several slides 
showed bulletin boards depicting the SMART learner theme. Participants appeared very 
attentive during the slide presentation; one participant, in fact, grinned widely throughout. 
Attendees applauded Dottie loudly as she concluded” (Howley-Rowe, 1999a). 

When participants at the November 1998 rally were asked for their assessment of 
presentations made by network schools on the final evaluation form, 27 of the 35 respondents made 
unambiguously positive comments. These included, “Tinder [Elementary] was great!,” “I thought 
this was wonderful and I always enjoy hearing from other schools,” “loved talking with Fred! He 
always offers so much!” and “always the best! Dottie and Fred did great!” (Howley-Rowe, 1999a). 

At the 1999 Scholars colloquium at Mountain Lake Resort, Virginia, one Scholar discussed 
the impact working with Dottie and other network participants had made, noting that networking for 
him meant that “if I face something difficult, I stop and think, ‘I could call Dottie or Evelyn.’ It 
calms me down ... to know they’re out there.” Later, this Scholar added, “Some of you in this room 
have really become my heros. I think, ‘What would Dr. Baldwin do?”’ He continued, noting a 
characteristic he associated with each of several other Scholars, describing how they peopled his 
mind during difficult moments at school. 

With the assistance of parents at the school, Tinder staff developed a video about their 
experience in the Quest network. Ultimately, this production was viewed by all regional laboratory 
directors and U.S. Department of Education staff at a meeting in September 1999 in Washington, 
D.C. In addition, principal Dottie Baldwin was a featured presenter at the October 1999 AEL Board 
meeting. There, she received an award in recognition of her continuous improvement efforts. 
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FINDINGS 

Innovation Configuration Checklist 

Quest staff developed an Innovation Configuration Checklist (ICC) specifying the essential 
components of the project and variations thereof. The ICC was administered to Quest leadership 
team members at the February 2000 rallies for elementary and high school networks to gather 
participants’ assessments of the degree to which their schools had been involved in the project. With 
a Cronbach alpha of .78, the overall scale possessed sufficient reliability for this administration. 
Items 1-7 provide four variations for respondents to select among; item 8 provides 3 options. For 
purposes of analysis, items 1 -7 were converted to a 4-point Likert-type scale, with 3 representing the 
most ideal variation of components and 0 representing the least satisfactory variation. Similarly, item 
8 was converted to a 3-point Likert-type scale. The overall scale score ranges from 0 to 23. 

Three Tinder Quest team members completed the ICC. The overall scale score of 1 8.00 (SD 
.61) indicates that the respondents felt their school had approximated the Quest specifications fairly 
well. 



Table 2 

Innovation Configuration Checklist Descriptive Statistics 



Abbreviated Items 


N 


Mean 


SD 


Quest School Leadership Team 


3 


3.00 


.00 


Administrative Support 


3 


2.33 


1.15 


Participation in Network Events 


3 


3.00 


.00 


Participation in Co-ventures in Learning 


2 


2.00 


1.41 


Involvement with Other Quest Schools 


3 


2.00 


1.73 


School Improvement/Action Research 


3 


2.67 


.58 


Change in Schoolwide View of School Improvement 


2 


1.00 


.00 


Engagement in Related School Improvement Efforts 


3 


2.00 


.00 


Scale 


3 


18.00 


.61 



The Tinder Quest team members gave a mean rating of 3.00 (SD .00) to the first ICC 
component, indicating that their school leadership team was inclusive with administrator, teacher, 
and parent membership; had been fairly stable over time; and had assumed active school leadership 
in taking the quest for continuous improvement back to the school community. The standard 
deviation indicates that respondents were in complete agreement about this. Also receiving a high 
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mean rating was the category of participation in co-ventures in learning (3.00, SD .00), in which a 
wide cross-section of the school community had participated. Again, respondents were in total 
agreement. Moreover, with a mean of 2.00 (SD .00), respondents unanimously indicated that their 
school had been actively engaged in other school improvement efforts, the results of which has been 
recognizable. 

Respondents were in somewhat less agreement about the extent of administrative support for 
involvement in Quest. With a mean of 2.33 (SD 1.15), Tinder participants felt that one or more of 
their school’s administrators had been members of the leadership team and had been occasionally 
involved in Quest activities. Respondents also believed that a limited number of their school 
community-primarily administrators, teachers, and staff-had participated in a Quest co-venture 
(2.00, SD 1 .41), although the standard deviation — the second largest — indicates that there was some 
disagreement about this. 

Involvement with other schools also received a mean rating of 2.00, again with a large 
standard deviation (SD 1.73), representing the most diverse set of responses on the instrument. The 
mean indicates that respondents thought individuals from their school had visited another Quest 
school or had hosted a visit by a Quest school to their campus. 

With a mean of 2.67 (SD .58), the presence of school improvement or action research 
projects at Tinder was rated fairly highly. This rating suggests that Tinder had implemented a school 
improvement project in connection with their involvement in Quest and collected data to 
demonstrate its impact on student learning. 

Least highly rated was the item concerning changes in the schoolwide view of school 
improvement (1.00, SD .00). This rating indicates that respondents unanimously thought members 
of the Quest leadership team had been affected by the Quest approach to continuous improvement. 
However, the rating also indicates that respondents did not believe that a wide cross-section of the 
school had been significantly impacted by the Quest approach, nor had they become committed to 
work on one or more of the framework components. 

Reflective Assessment Findings 

Six Quest team members completed the Reflective Assessment questionnaire. This 
instrument asks respondents to rate their school’s convergence with a description of a school 
representing a “ 1 00” on a scale of 0- 1 00 in increments of 1 0 with respect to each component from 
the Quest framework from continuous improvement. They are then requested to cite evidence or 
examples supporting their rating and describe the ways, if any, in which Quest made an impact on 
their school’s development with regard to the component under consideration. Finally, respondents 
are asked what factors other than Quest have influenced their school’s development. 
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Table 3 

Reflective Assessment Descriptive Statistics 
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Quest Framework Component 


N 


Mean 


SD 


Strengthening the Learning Culture 


6 


87.50 


11.73 


Broadening the Learning Community 


6 


85.00 


10.49 


Sharing Leadership 


6 


77.50 


8.80 


Shared Goals for Learning 


6 


93.33 


5.16 


Assessing and Demonstrating Learning 


6 


94.17 


4.92 


Enabling SMART Learners 


6 


93.33 


5.16 



Quest team members assigned the highest mean rating (94.17, SD 4.92) to the degree to 
which Tinder approximated the description of a school excelling with regard to assessing and 
demonstrating learning. This item also had the smallest standard deviation, suggesting that team 
members were in some agreement about their school’s status in terms of the component. 

Three of the six respondents provided multiple replies when asked to cite evidence of their 
school’s commitment to assessing and demonstrating learning. Noted twice was Tinder’s 
administration of the state mandated standardized test early in the school year and again at the end 
of the year. As one staff member put it, “We go beyond our district’s call for assessing our students 
on the CAT5, for we pretest and posttest to measure concept achievements.” 

Also noted in two instances was the lack of time provided staff for reflection on assessment 
results, despite the school’s efforts to produce such data. On the other hand, one respondent reported 
providing opportunities to allow students to consider the implications of their performance. 

Multiple kinds of student data are generated, according to one staff member. Two 
respondents also indicated that student data of various kinds are shared with school entities such as 
the site based decision making council. 

Professional development is planned around areas of need suggested by student data, reported 
one Tinder respondent. Another noted that the action research on math instruction and achievement 
had been inspired by staff analysis of math scores over time; Quest staff introduced Tinder to 
Eisenhower Math/Science Consortium staff who collaborated in the effort. 

When asked in what ways Quest made an impact on the school’s growth in terms of 
assessment, one team member offered a reply with multiple themes. Two respondents mentioned the 
ideas they had acquired during project events and interactions. Another staff member added that 
Quest staff had provided research and opportunities for sharing ideas. 
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According to one respondent, participation in the Quest network had enabled Tinder staff to 
“look at the successes and failures of others and gain knowledge and inspiration from them.” 
Another team member reported that the project had convinced school staff of the importance of 
reflection, writing, “We have a much stronger sense of the critical nature of providing time for 
reflection on self study. We need to take time to think.” 

Student-led conferences had contributed to the school’s approach to assessment, said one 
respondent. Another noted that the action research undertaken in conjunction with Quest around 
math had made an impact. 

Only one respondent replied to the query soliciting information about other factors 
influencing Tinder’s growth in terms of assessing and demonstrating learning. This respondent noted 
that the principal’s involvement with and knowledge of assessment innovations had been a “ strong 
reason for our strength in this area.” 

Receiving the lowest mean rating was the component of shared leadership (77.50, SD 8.80). 
Four respondents gave multiple answers when asked to cite evidence of the degree to which their 
school approximated the description of a school committed to sharing leadership. Although two team 
members reported many forums allowing for faculty, staff, parent, and student input, two 
respondents expressed dissatisfaction with the extent to which leadership is shared at Tinder. Said 
one, “I feel as though the concerns I have represented from our faculty have not been given the 
thought that a shared leader could [give them].” 

Two other respondents were more ambivalent, noting that concerns were addressed but that 
school administrators retained the right to make decisions “they deem best.” Another staff member 
noted being too busy to stay apace of opportunities for shared leadership. 

Several challenges to sharing leadership were reported. The principal shared that 
communication with staff was a priority for her, but that “I still fall short in this area.” Another 
respondent felt that most school community members — “the silent majority” — were not as 
committed to “be[ing] heard” as were others. Yet another felt that time for participating in shared 
decision making was a concern. 

On the other hand, one respondent reported that teachers “take an active role in school 
leadership through committees and council work . . . Student leadership is encouraged and very 
active. It is mentored through the leadership program and student council.” 

Only three team members replied to the query about what ways Quest had made an impact 
on shared leadership at Tinder. Two comments indicated that Quest had reinforced the importance 
of including student voice in decision making and leadership. As one respondent said, “We have 
learned through student-led conferences to really listen to our children ... To listen to what they say, 
how they say it, and sometimes what they don’t say.” A third respondent reported that Quest had 
increased awareness of the “need to keep everyone involved” and had provided “tools to do so.” 
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Professional Learning Community Findings 

In March 2000, Tinder staff completed a post-test of the School Professional Staff as 
Learning Community survey developed by Hord (1997; Meehan, Orletsky, & Sattes, 1997). This 
instrument consists of five main subsections: shared leadership, shared visions, collective creativity, 
peer review, and supportive conditions and capacities (Cowley, 1999). Subsections contain several 
individual items respondents are asked to rate using a 5-point Likert-type scale, with anchor points 
of low (1) and high (5). However, the field test of the survey revealed that it measures one overall 
construct rather than five distinct factors (Meehan, Orletsky, & Sattes, 1997). This construct could 
be described as the extent to which school staff constitute a supportive professional learning 
community. Therefore, an overall score is calculated for the instrument; the higher the score, the 
more respondents feel their school is a positive learning community. The instrument contains 17 
items, and the overall score may range from 17-85 points. 

The mean post-test score was somewhat lower than the February 1998 pre-test score. The 
pre-test score (65.00, SD 12.18) indicated that Tinder faculty believed they were a professional 
learning community in many ways, although they had room to grow. The post-test score of 62.41 
(SD 10.57) was 2.59 points lower than the pre-test score. 

Table 4 

t Test Results for the Professional Learning Community Instrument 



Administration of Professional Learning 
Community Instrument 


N 


Mean 


SD 


t 


Sig. 


Pre-test Total 


22 


65.00 


12.18 


.696 


.490 


Post-test Total 


17 


62.41 


10.57 



A t test of total Professional Learning Community pre- and post-test scores revealed that the 
differences were likely not attributable to chance. It should be noted that the assumptions of the t test 
were violated in this study. The sample was not random, nor was it assumed that the data were 
drawn from a normally distributed population. Phillips (1 982) contends, however, that “since those 
assumptions now appear to be far less important than originally thought, the recent trend toward 
increasing use of distribution-free tests is currently being reversed” (p. 139). Likewise, Glass and 
Hopkins (1984) report research suggesting that violation of the assumptions of normality and 
homogeneity of variance has little impact upon the robustness of t tests. For these reasons, Quest 
staff chose to use the t test to explore the statistical significance of pre- and post-test differences on 
the Professional Learning Community instrument. 

The t value of .696 was not significant at the .05 level of statistical significance. In other 
words, the difference in pre- and post-test scores were more likely due to chance than to a 
substantive change in Tinder as a learning community. 
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California Achievement Tests, Fifth Edition 

The California Achievement Tests, fifth edition (CAT5), are a battery of tests measuring 
students’ development in reading, language, math, spelling, science, social studies, and study skills. 
Normed in 1991 and published in 1992, the CAT5 is available as a Complete Battery and a shorter, 
less reliable Survey Battery. The Complete Battery is administered to students in grades 2 through 
6 at Tinder Elementary. According to Nitko (1998) and McMorris, Liu, and Bringsjord (1998), the 
test possesses adequate validity and reliability; hence, CAT5 scores serve as useful indicators of 
student achievement. 

Tinder CAT5 scores from spring administrations of the test were analyzed for 1997, 1998, 
and 1999. Data for spring 2000 were not available at the time of this writing. For each grade level 
tested, mean national percentile scores were generated for each CAT5 area subtest. Total reading, 
total language, total math, and total battery scores were also calculated. CAT5 descriptive statistics 
were generated for the whole school, as well. 

For Tinder as a whole, national percentile scores dipped somewhat during the 1998 school 
year (see Table 5). Flowever, scores for many CAT5 sections were higher in 1999 than in 1997, 
including reading vocabulary, reading comprehension, language mechanics, total language, math 
concepts and application, the total reading and math battery, and social studies. Scores on the total 
math battery declined a bit in 1998, but closely approximated 1997 levels in 1999. Mean national 
percentiles increased steadily between 1 997 and 1 999 on the total reading battery. Science percentile 
scores remained stable during 1 997 and 1 998, but then increased in 1 999. Differences between 1 997 
and 1999 mean national percentile scores ranged from .75 percentile points for the math concepts 
and application section of the CAT5 to 4.44 on the science section. 

Scores for other sections on the CAT5 decreased somewhat. Mean national percentiles were 
a bit lower in 1 998 than they had been in 1 997 for the language expression and spelling sections of 
the test. Scores climbed slightly for these sections in 1999 but did not reach their 1997 levels. Math 
computation scores declined somewhat each year, although only by 1.79 percentile points. 

CAT5 mean national percentile scores for second grade show steady declines between 1 997 
and 1 999 in all areas except math concepts and application, total math battery, total core area battery, 
word analysis, science, and social studies (see Table 6). In these six sections, scores fell between 
1997 and 1998, and then rose somewhat in 1999, although not to their previous levels. Scores 
dropped from between 4.07 (math computation) percentile points to 14.37 (science) in CAT5 areas 
showing steady decline. 
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Table 5 

Tinder Elementary CAT5 Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N 


Mean 


SD 


Reading Vocabulary 


1997 


488 


63.47 


28.16 




1998 


532 


62.56 


27.50 




1999 


505 


64.75 


25.93 


Reading Comprehension 


1997 


487 


65.51 | 


29.90 




1998 


532 


64.52 


29.05 




1999 


505 


66.72 


28.93 


Reading Total 


1997 


487 


65.75 


28.80 




1998 


532 


66.38 


40.64 




1999 


505 


67.49 


27.46 


Language Mechanics 


1997 


488 


69.49 


28.34 




1998 


532 


68.28 


27.99 




1999 


503 


71.29 


27.06 


Language Expression 


1997 


488 


66.09 


47.68 




1998 


532 


62.59 


30.02 




1999 


504 


65.28 


29.97 


Language Total 


1997 


488 


68.08 


29.89 




1998 


532 


66.74 


29.21 




1999 


503 


69.54 


28.40 


Math Computation 


1997 


490 


66.46 


28.84 




1998 


532 


65.93 


27.34 




1999 


504 


64.67 


28.75 


Math Concepts and Application 


1997 


489 


72.16 


27.25 




1998 


532 


70.49 


27.59 




1999 


504 


72.91 


26.18 


Math Total 


1997 


489 


71.96 


27.71 




1998 


532 


70.47 


27.66 




1999 


504 


71.51 


27.48 


Total Battery 


1997 


487 


70.15 


29.01 




1998 


532 


68.72 


28.47 




1999 


503 


71.44 


27.37 


Spelling 


1997 


488 


64.56 


43.80 




1998 


532 


64.20 


44.40 




1999 


505 


61.62 


30.36 


Science 


1997 


488 


| 66.65 


27.61 




1998 


531 


66.40 


26.54 




1999 


504 


71.09 


25.84 


Social Studies 


1997 


487 


67.59 


27.25 




1998 


531 


65.95 


27.46 




1999 


504 


69.20 


25.85 
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Table 6 

Tinder Elementary Grade 2 CAT5 Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N 


Mean 


SD 


Reading Vocabulary 


1997 


101 


66.83 


25.81 




1998 


113 


59.02 


27.10 




1999 


99 


58.59 


26.51 


Reading Comprehension 


1997 


100 . 


72.22 


28.35 




1998 


113 


67.05 


31.32 




1999 


99 


66.43 


30.27 


Reading Total 


1997 


100 


72.45 


27.33 




1998 


113 


71.44 


70.03 




1999 


99 


65.25 


29.72 


Language Mechanics 


1997 


101 


75.12 


23.30 




1998 


113 


70.26 


26.03 




1999 


98 


72.47 


26.98 


Language Expression 


1997 


101 


72.22 


26.60 




1998 


113 


66.35 


28.41 




1999 


99 


65.21 


29.94 


Language Total 


1997 


101 


76.00 


24.09 




1998 


113 


69.84 


26.47 




1999 


98 


70.27 


28.09 


Math Computation 


1997 


101 


72.24 


24.77 




1998 


113 


70.00 


25.63 




1999 


99 


68.17 


28.30 


Math Concepts and Application 


1997 


101 


78.68 


23.54 




1998 


113 


71.10 


28.51 




1999 


99 


76.39 


25.63 


Math Total 


1997 


101 


78.67 | 


23.46 




1998 


113 


72.96 


27.40 




1999 


99 


75.72 


26.91 


Total Battery 


1997 


100 


78.31 


24.36 




1998 


113 


71.00 


28.77 




1999 


98 


72.82 


27.14 


Word Analysis 


1997 


101 


61.54 


26.29 




1998 


113 


63.46 


27.85 




1999 


99 


60.30 


29.31 


Spelling 


1997 


101 


73.16 


75.07 




1998 


113 


64.32 


55.14 




1999 


99 


58.79 


30.29 


Science 


1997 


101 


77.40 


25.29 




1998 


113 


69.15 


26.69 




1999 


99 


74.44 


25.92 


Social Studies 


1997 


101 


75.30 


26.55 




1998 


113 


70.64 


30.25 




1999 


99 


73.13 


26.11 
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Table 7 

Tinder Elementary Grade 3 CATS Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N ■ 


Mean 


SD 


Reading Vocabulary 


1997 


106 


65.03 


29.74 




1998 


115 


66.57 


29.41 




1999 


103 


69.48 


26.26 


Reading Comprehension 


1997 


106 


62.39 


30.47 




1998 


115 


65.22 


29.70 




1999 


103 


70.33 


29.27 


Reading Total 


1997 


106 


64.57 


30.04 




1998 


115 


67.19 


29.90 




1999 


103 


71.60 


27.96 


Language Mechanics 


1997 


106 


62.66 


30.26 




1998 


115 


60.01 


30.47 




1999 


103 


65.82 


28.75 1 


Language Expression 


1997 


106 


66.83 


32.66 




1998 


115 


65.60 


31.68 




1999 


103 


68.11 


32.01 


Language Total 


1997 


106 


65.88 


31.94 




1998 


115 


65.01 


31.38 




1999 


103 


68.09 


30.47 


Math Computation 


1997 


107 


62.81 


27.82 




1998 


115 


68.59 


27.02 




1999 


103 


68.62 


26.03 


Math Concepts and Application 


1997 


107 


69.49 


27.20 




1998 


115 


69.62 


28.70 




1999 


103 


74.15 


22.85 


Math Total 


1997 


107 


69.37 


27.69 




1998 


115 


71.28 


28.80 




1999 


103 


74.28 


24.39 


Total Battery 


1997 


106 


67.74 


30.50 




1998 


115 


69.46 


30.20 




1999 


103 


72.81 


27.74 


Word Analysis 


1997 


106 


59.14 


| 30.90 




1998 


115 


62.18 


30.55 




1999 


103 


65.09 


29.07 


Spelling 


1997 


106 


57.77 


| 33.48 




1998 


115 


60.46 


32.74 




1999 


103 


64.02 


29.25 


Science 


1997 


106 


66.20 


26.82 




1998 


115 


67.90 


26.09 




1999 


103 


73.60 


24.45 


Social Studies 


1997 


105 


68.18 


26.06 




1998 


115 


66.40 


25.38 




1999 


103 


73.10 


24.28 
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Tinder third grade CAT5 scores reveal a pattern very different from that found in second 
grade scores (see Table 7). Third grade mean national percentile scores increased consistently 
between 1997 and 1999, with the exception of language mechanics, language expression, total 
language battery, and social studies sections. In these areas, scores declined somewhat in 1998, 
then exceeded 1997 achievement levels in 1999. Increases for all areas ranged from 1.28 
percentile points in language expression to 7.94 points in reading comprehension. 

All 1999 fourth grade CAT5 scores were higher in 1999 than they had been in 1997, most 
increasing steadily (see Table 8). Mean national percentiles for math computation, math concepts 
and application, total math battery, spelling, study skills, and social studies rose in 1 998 and then 
declined somewhat in 1 999, although remaining higher than in 1 997. These scores increased between 
3.72 percentile points (science) and 9.32 points (total math battery) between 1997 and 1998. 
Declines for these areas ranged from 2. 1 1 percentile points (math concepts and application) to 5.29 
percentile points (spelling). For those sections in which Tinder fourth graders showed consistent 
improvement between 1997 and 1999, mean national percentile increases ranged from 4.29 
percentile points (reading comprehension) to 8.22 (total language battery). 

Overall, Tinder fifth grade CAT5 scores increased in most areas (see Table 9). Reading 
vocabulary, total reading battery, language mechanics, science, and social studies all increased 
steadily between 1997 and 1999. Increases for these areas ranged from .37 percentile points 
(language mechanics) to 15.81 (science). In another five CAT5 sections, mean national percentiles 
fell in 1998, then rose in 1999. In addition, scores for all five areas in 1999 were higher than 1997 
scores (reading comprehension, total language battery, total core area battery, spelling, and study 
skills). 



Two areas showed consistent decline over the three years, however. Math computation scores 
in 1999 were 7.99 percentile points lower than they had been in 1997, and total math battery scores 
were lower by 4.82 percentile points. In two other sections, scores dipped in 1998 before rising in 
1999, although not reaching 1997 levels. Language expression scores were 13.96 percentile points 
lower in 1998 than in 1997; in 1999, mean national percentile achievement had increased by 8.48 
points. In 1998, math concepts and application scores dropped by 5.57 percentile points then rose 
by 4.09 points in 1999. 

All 1 999 CAT5 mean national percentile scores were lower in 1 999 than they had been in 
1 997 (see Table 1 0). In all but the spelling section of the test, scores dropped in 1 998, then increased 
in 1 999, many nearing 1 997 achievement levels. Declines in 1 998 ranged from 3 .03 percentile points 
(language mechanics) to 12.92 points (math computation). Scores rose again in 1999, ranging from 
a 1.6 percentile point for math concepts and application to 8.36 for math computation. 




C t 

CO 



Table 8 

Tinder Elementary Grade 4 CAT5 Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N 


Mean 


SD 


Reading Vocabulary 


1997 


102 


63.01 


24.88 




1998 


115 


66.28 


24.73 




1999 


100 


67.63 


23.50 


Reading Comprehension 


1997 


102 


64.56 


30.03 




1998 


115 


68.32 


24.81 




1999 


100 


68.85 


29.94 


Reading Total - , 


1997 


102 


64.84 


27.32 




1998 


115 


69.09 


23.77 




1999 


100 


70.27 


26.56 


Language Mechanics 


1997 


102 


63.94 


31.55 




1998 


115 


68.21 


27.13 




1999 


100 


71.08 


25.26 


Language Expression 


1997 


102 


55.74 


31.42 




1998 


115 


61.94 


28.01 




1999 


100 


63.15 


30.22 


Language Total 


1997 


102 


60.54 


32.71 




1998 


115 


66.34 


27.85 




1999 


100 


68.76 


27.77 


Math Computation 


1997 


102 


61.09 


32.32 




1998 


115 


69.74 


27.07 




1999 


100 


66.13 


27.36 


Math Concepts and Application 


1997 


102 


69.44 


28.70 




1998 


115 


77.00 


23.52 




1999 


100 


74.89 


27.02 


Math Total 


1997 


102 


67.00 


30.34 




1998 


115 


76.32 


24.36 




1999 


100 


73.38 


26.63 


Total Battery 


1997 


102 


64.30 


30.62 




1998 


115 


71.66 


24.59 




1999 


100 


72.32 


27.35 


Spelling 


1997 


102 


59.28 


32.13 




1998 


115 


68.38 


29.21 




1999 


100 


63.09 


31.11 


Study Skills 


1997 


102 


67.16 


25.94 




1998 


115 


75.59 


20.23 




1999 


100 


73.19 


22.00 


Science 


1997 


102 


66.45 


27.20 




1998 


115 


70.81 


24.29 




1999 


100 


70.84 


25.94 


Social Studies 


1997 


102 


65.79 


29.52 




1998 


115 


69.51 


28.70 




1999 


100 


67.04 


29.34 
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Table 9 

Tinder Elementary Grade 5 CAT5 Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N 


Mean 


SD 


Reading Vocabulary 


1997 


81 


62.28 


32.65 




1998 


99 


63.33 


27.17 




1999 


108 


68.37 


24.38 


Reading Comprehension 


1997 


81 


65.46 


30.05 




1998 


99 


63.13 


29.01 




1999 


108 


68.31 


27.32 


Reading Total 


1997 


81 


64.41 


29.34 




1998 


99 


64.93 


27.79 




1999 


108 


70.03 


25.28 


Language Mechanics 


1997 


81 


74.16 


26.30 




1998 


99 


74.17 


26.54 




1999 


108 


74.53 


26.76 


Language Expression 


1997 


81 


72.65 


94.83 




1998 


99 


58.69 


30.49 




1999 


108 


67.18 


27.58 


Language Total 


1997 


81 


68.88 


28.76 




1998 


99 


67.20 


30.22 




1999 


108 


71.93 


27.24 


Math Computation 


1997 


81 


55.48 


27.52 




1998 


99 


53.62 


27.13 




1999 


108 


47.79 


25.04 


Math Concepts and Application 


1997 


81 


72.30 


26.77 




1998 


99 


66.73 | 


27.36 




1999 


108 


70.82 


24.99 


Math Total 


1997 


81 


67.13 


28.21 




1998 


99 


62.56 


28.01 




1999 


108 


62.31 


27.01 


Total Battery 


1997 


81 


69.18 


28.23 




1998 


99 


66.18 


28.75 




1999 


108 


70.48 


26.28 


Spelling 


1997 


81 


65.87 


27.74 




1998 


99 


64.41 


64.81 




1999 


108 


66.41 


29.57 


Study Skills 


1997 


81 


68.34 


24.24 




1998 


99 


65.17 


26.37 




1999 


108 


70.67 


22.49 


Science 


1997 


81 


57.01 


26.65 




1998 


99 


| 64.13 


26.73 




1999 


108 


72.82 


25.17 


Social Studies 


1997 


81 


64.45 


26.72 




1998 


99 


65.61 


25.45 




1999 


108 


69.91 


24.07 
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Table 10 

Tinder Elementary Grade 6 CAT5 Mean National Percentile Scores, 1997-1999 
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CAT5 Section 


Administration 

Year 


N 


Mean 


SD 


Reading Vocabulary 


1997 


98 


59.77 


28.01 




1998 


90 


56.28 


28.10 




1999 


95 


58.91 


27.25 


Reading Comprehension 


1997 


98 


63.06 


30.10 




1998 


90 


57.13 


29.49 




1999 


95 


59.05 


26.99 


Reading Total 


1997 


98 


62.25 


29.46 




1998 


90 


57.11 


28.80 




1999 


95 


59.55 


26.46 


Language Mechanics 


1997 


98 


72.98 


26.99 




1998 


90 


69.95 


27.91 




1999 


94 


72.56 


27.09 


Language Expression 


1997 


98 


64.30 


29.86 




1998 


90 


59.15 


31.43 




1999 


94 


62.32 


30.25 


Language Total 


1997 


98 


69.45 


29.25 




1998 


90 


65.06 


30.51 




1999 


94 


68.43 


28.75 


Math Computation 


1997 


99 


79.04 


25.55 




1998 


90 


66.12 


27.06 




1999 


94 


74.48 


30.08 


Math Concepts and Application 


1997 


98 


71.10 


29.09 




1998 


90 


66.65 


29.08 




1999 


94 


68.20 


29.99 


Math Total 


1997 


98 


77.05 


26.80 




1998 


90 


67.54 


28.33 




1999 


94 


72.62 


30.74 


Total Battery 


1997 


98 


71.32 


29.29 




1998 


90 


63.94 


29.89 




1999 


94 


68.68 


28.77 


Spelling 


1997 


98 


67.44 


27.02 




1998 


90 


63.26 


28.39 




1999 


95 


54.98 


30.89 


Study Skills 


1997 


98 


70.25 


27.17 




1998 


90 


59.33 


27.43 




1999 


94 


62.80 


25.27 


Science 


1997 


98 


64.22 


28.78 




1998 


89 


57.77 


27.81 




1999 


94 


63.11 


26.75 


Social Studies 


| 1997 


98 


63.48 


26.06 




1998 


89 


55.19 


24.37 




1999 


94 


62.27 


24.04 
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Tests of statistical significance were run to discern any significant increases in student 
performance over several years of Tinder’s participation in Quest. Via analyses of variance 
(ANOVA), only three statistically significant increases at the .05 level in CAT5 national percentile 
scores were found within tested grades. Only one statistically significant increase was located within 
the overall school data: Science national percentile scores were higher in 1 999 than in 1 997 and 1 998 
(F ratio of 4.978, F probability of .01). These findings are presented in Table 11. 



Table 11 

CAT 5 ANOVA Results 











Significant 


CAT5 Subsection 


df 


F ratio 


F 


Differences by 








prob. 


Test Year 


4 th Grade Math Total Battery 


2 


3.291 


.04* 


1999 > 1997 




314 








4 th Grade Study Skills Subtest 


2 


3.865 


.02* 


1998 > 1997 




314 








5 th Grade Science Subtest 


2 


8.646 


.00* 


1999 > 1998, 




285 






1997 


Whole School Science Subtest 


2 


4.978 


.01* 


1999 > 1998, 




1520 






1997 



* Statistically significant at the .05 level. 



Statistical significance alone, however, does not indicate the meaningfulness of findings; 
rather, it indicates the rareness of findings. The calculation of effect size allows the conversion of 
statistically significant results into the standard deviation metric, providing a better analysis of 
practical significance. 

With an effect size of d= .14, improvements in fourth grade total math battery scores were 
small. Similarly, fourth grade study skills scores had a small effect size of d = . 1 6. The largest effect 
size of d= .25 for increases in fifth grade science scores between 1997 and 1999 was nonetheless 
small, according to conventional classifications of the meaning of effect sizes (Cohen, 1988). With 
an effect size of d = .08, improvements in science scores for the whole school were quite small. 

In sum, Tinder students’ achievement showed some gains, particularly in the third and fourth 
grade trends between 1997 and 1999. But overall, little statistically significant change took place, 
and declines were found as well, particularly in the second and sixth grades. Where statistical 
significance was found, effect sizes revealed that such improvements were not substantial. 

Quest appears to have had little bearing on standardized student achievement at Tinder, 
although an improvement trend was noted in the third and fourth grades, and somewhat in the fifth 
grade. This relative lack of impact is not surprising, however, given that many of the school’s efforts 
undertaken through Quest were not squarely and clearly directed at increasing achievement scores, 
as school staff found other foci to be of greater relevance. 
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Test of Cognitive Skills 

The Test of Cognitive Skills, second edition (TCS/2), published by CTB/Macmillan/McGraw- 
Hill, is administered yearly to Tinder students in grades 2 through 6. This standardized instrument is 
intended to assess general academic ability via four subtests: one verbal; two nonverbal, which are 
combined for a composite score; and one memory. The nonverbal components of the instrument 
include items on sequences and analogies. A total score, or Cognitive Skills Index, is also yielded 
for the entire test. The TCS/2 was normed in 1991 and has not been normed since. Scores are thereby 
presented as national percentiles by grade. According to Kamphaus (1998), the instrument was 
developed with sufficient rigor to yield meaningful measures and “sets a high standard with respect 
to diligent and thoughtful test development procedures for academic ability testing.” 

The TCS/2 is administered to groups of students and takes approximately one hour to 
complete. Kamphaus’s review of the instrument indicates that administration guidelines for 
examiners are “clear and succinct” (1998, p. 1 027). He also suggests that test items are unambiguous 
and well organized in response booklets. 

The TCS/2 was administered to students in grades 2-6 in 1996, 1997, and 1998. It was 
administered to students in grades 3-6 in 1 999. Higher 1999 scores should be interpreted with some 
caution; they may be an artifact of the somewhat older students in the 1999 sample. 

As shown in Table 12, mean national percentile by grade scores on the TCS/2 nonverbal 
composite rose steadily between 1 996 and 1 999. The mean difference between 1 996 and 1 999 scores 
was 6.86 percentile points. Consistent increases were not observed, however, with regard to the 
memory subtest, although 1999 means were higher than 1996 and 1998 scores (see Table 13). 

Mean national percentiles by grade increased consistently between 1996 and 1999 on the 
verbal subsection of the TCS/2, rising from 48.52 (SD 31.15) to 55.74 (SD 31.08). The mean 
difference in scores was 7.22 percentile points. Table 14 displays these data. 

As seen in Table 15, mean national percentile by grade scores grew from 52.64 (SD 29.30) 
in 1996 to 60.25 (SD 29.16) in 1999. This represents a mean difference of 7.61 percentile points. 

Table 12 

TCS/2 Nonverbal Composite Mean National Percentiles by Grade 



TCS/2 Nonverbal Composite Administration 


N 


Mean National 
Percentile by 
Grade 


SD 


1996 


502 


54.91 


29.61 


1997 


536 


57.54 


29.62 


1998 


497 


60.94 


28.79 


1999 


400 


61.77 


28.22 
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Table 13 

TCS/2 Memory Subtest Mean National Percentiles by Grade 
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TCS/2 Memory Subtest Administration 


N 


Mean National 
Percentile by 
Grade 


SD 


1996 


502 


53.20 


30.00 


1997 


536 


56.26 


31.28 


1998 


497 


54.49 


31.66 


1999 


402 


55.09 


30.90 



Table 14 

TCS/2 Verbal Subtest Mean National Percentiles by Grade 



TCS/2 Verbal Subtest Administration 


N 


Mean National 
Percentile by 
Grade 


SD 


1996 


502 


48.52 


31.15 


1997 


536 


50.33 


31.65 


1998 


497 


52.40 


31.25 


1999 


401 


55.74 


31.08 



Table 15 

TCS/2 Total Cognitive Skills Index Mean National Percentiles by Grade 



TCS/2 Total Cognitive Skills Index Administration 


N 


Mean National 
Percentile by 
Grade 


SD 


1996 


502 


52.64 


29.30 


1997 


536 


55.72 


30.28 


1998 


497 


58.18 


29.36 


1999 


400 


60.25 


29.16 



ANOVAs were conducted to compare the variance between and within groups of respondents 
based upon the testing year. Three statistically significant differences were located at the .05 level 
of statistical significance. Tukey HSD post hoc tests were conducted to determine among which of 
the four testing years the statistically significant differences were located. Statistically significant 
ANOVA results are presented in Table 16. 



ERIC 
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As noted earlier with regard to the use of t tests to examine differences in scores on the 
Professional Learning Community instrument, the assumptions of ANOVA were violated in this 
study. The sample was not random, nor was it assumed that the data were drawn from a normally 
distributed population or that the samples had homogenous variances. Despite arguments that such 
violations do not compromise the robustness of ANOVA to the extent previously thought, all 
findings should be interpreted with caution. 



Table 16 

TCS/2 ANOVA Results 



TCS/2 


df 


F-ratio 


F probability 


Significant 
Differences by 
Test Year 


Nonverbal Composite 


3 

1931 


5.57 


* 

o 

o 


1996<1 998, 1999 


Memory Subtest 


3 

1933 


0.87 


.46 


N/A 


Verbal Subtest 


3 

1932 


4.36 


o 

o 

* 


1996<1 998, 1999 


Total Cognitive Skills Index 


3 

1931 


5.65 


o 

o 

* 


1 996<1 998, 1999 



* Statistically significant at the .05 level. 



These data are particularly interesting given changes in Tinder’s attendance zone, which 
combined suburban subdivisions with a public housing project. Unfortunately, student demographic 
data are not available to disaggregate findings. 

With an F ratio of 5.57 (F probability of .00), mean national percentiles by grade were 
statistically significantly higher in 1998 and 1999 than in 1996, the year prior to Tinder’s 
participation in Quest. Mean national percentiles by grade on the verbal subtest were also statistically 
significantly higher in 1998 and 1999 than in 1996 (F ratio of 4.36, F probability of .00). Finally, 
with an F ratio of 5.65 and an F probability of .00, mean national percentiles by grade increased by 
a statistically significant amount on the total TCS/2: As with the verbal and nonverbal subsections, 
scores were higher in 1998 and 1999 than in 1996. 

Effect sizes for the three statistically significant TCS/2 increases were calculated to determine 
how sizeable they were. With d = .09, national percentiles by grade on the nonverbal component of 
the instrument were higher by only .09 of a standard deviation between 1996 and 1999. The effect 
sizes for scores on the verbal subtest ( d = .08) and the total Cognitive Skills Index (d= .09) were 
similarly small. 

In sum, it appears that national percentile increases on the TCS/2 at Tinder Elementary over 
the course of the school’s participation in Quest were likely not due to chance or sampling error. 
However, the increases themselves were relatively small, accounting for approximately only a tenth 
of a standard deviation. 
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ANOVAs were also conducted to determine where statistically significant differences lay 
within each grade level tested (see Table 17). Second grade scores on the TCS/2 were statistically 
significantly higher in 1997 and 1998 than in 1996 on the nonverbal subtest (F-ratio of 5.57, F 
probability of .00). Although the ANOVA indicated that the increase in third grade scores on the 
nonverbal composite was likely not due to chance (F-ratio of 3.3 1, F probability of .02), Tukey HSD 
post hoc tests failed to locate any statistically significant differences among testing years. With an 
F-ratio of 3.79 and an F probability of .01, fourth grade scores were determined to be statistically 
significantly higher in 1999 than in 1996. 

Significant differences were found with respect to the verbal subtest of the TCS/2 in only the 
third grade scores. National percentiles by grade were statistically significantly higher in 1 999 than 
in 1996 (F-ratio of 3.64, F probability of .01). 

Statistically significant differences were located in the second, third, and fourth grades on 
the total Cognitive Skills Index. With an F-ratio of 3.20 (F probability of .04), scores on the total 
index were higher in 1 996 than in 1 997 for second graders completing the instrument. Third graders 
scored statistically significantly higher on the total index in both 1 998 and 1 999 than in 1 996 (F-ratio 
3.74, F probability of .01). Fourth grade national percentiles were statistically significantly greater 
in 1999 than in 1996 (F-ratio of 4.03, F probability of .01). 

Table 17 

TCS/2 Statistically Significant ANOVA Results by Grade 



TCS/2 


Grade 

Level 


df 


F-ratio 


F probability 


Significant 
Differences by 
Testing Year 


Nonverbal Composite 


2 


2 

299 


5.57 


.00* 


1996 < 1997, 
1998 


3 


3 

445 


3.31 


.02* 


No significant 
differences by 
year located 


4 


3 

385 


3.79 


.01* 


1996 < 1999 


Verbal Subtest 


3 


3 

445 


3.64 


.01* 


1996 < 1999 


Total Cognitive Skills Index 


2 


2 

299 


3.20 


.04* 


1996 < 1997 


3 


3 

445 


3.74 


.01* 


1996 < 1998, 
1999 


4 


3 

385 


4.03 


.01* 


1996 < 1999 



* Statistically significant at the .05 level. 
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No statistically significant differences in scores were located on the memory subtest of the 
TCS/2. In addition, no significant differences were discovered in fifth and sixth grade scores between 
1996 and 1999. 

Effect sizes were calculated to explore the practical significance of statistically significant 
findings within grades tested. Within grades, effect sizes were only slightly larger than those for the 
statistically significant increases for the entire school. The effect size for second grade nonverbal 
subtest scores was d= .19; for second grade total Cognitive Skills Index scores, d= .14. For both 
the verbal subtest and the total index, the effect size for third graders was d= .1 6. In terms of fourth 
grade scores, the effect size for the nonverbal subtest was d= .17 and d= .18 for the total Cognitive 
Skills Index. 

Effect sizes were therefore somewhat larger within grades two through four than for the 
entire school. However, even the highest effect size of d= .19 for second grade nonverbal subtest 
scores is conventionally considered small (Cohen, 1988). 

Focus Group and Individual Interview Findings 
Personal Impact of Quest 

Participants in individual interviews, a faculty focus group, and the Quest team discussion 
were asked what impact participation in Quest had engendered on the personal and professional 
level. Respondents offered 25 replies constituting 1 1 themes. One reply was coded twice as it 
contained two distinct themes. 

Mentioned five times by parent interviewees was the way in which Quest had enabled them 
to learn more about educational issues, processes, structures, and jargon. Said one such respondent, 
“I think my involvement in Quest has enabled me to become a lot more reflective about education. 
I have had the opportunity to be around educators which has allowed me to understand more closely 
all of the details involved in carrying out the day to day activities that are involved in teaching 
children. Uh, definitely the exposure to the literature in the educational field has been a very positive 
aspect of Quest for me.” Another commented, “Going to Quest and listening to these people and 
looking at the different research has given me what I feel at least are the beginnings of foundation 
of understanding to help people learn and of the educational process.” Such insight into the workings 
of the educational process is important because, as one interviewee put it, “It has helped me 
understand them and how they operate, and it's helped me to see their relationship with my child. 
It has also helped me see their relationship with each other and Dr. Baldwin and how that whole 
process works. I mean that's, that's a real mystery to parents, you know, how things get done in 
school.” 



Similarly, in two instances team members reported that the proximity to teachers afforded 
by inclusive Quest teams had enabled them to establish new relationships and gain insight into 
teachers’ work lives. As one indicated, “I think probably what Quest has allowed me to do is to get 
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into the mind set of the educator a little bit, not a lot, but a little bit, and to understand that they're 
people too. They have lives, they have things that they have to do, and you know, the needs that they 
have as far as their students are concerned. You know, so many . . . when I was a kid you'd look at 
your teacher and think, ‘They don't have a life . . . They stay at school.’” 

Focusing on how to share leadership in the school was noted in four instances. The principal, 
for example, reported that this issue had been of particular concern to her during Tinder’s tenure in 
Quest. She shared, “How do you really share leadership? Delegation of authority seems like a very 
easy kind of thing or delegation of responsibility. But it's more than that. It's being able, and this is 
. . . a tough thing to do I've found ... to step out of the role you're in to become a member, a 
colleague, and become a member of a problem solving group and say, ‘This is where we need to go. 
Now let's talk about.’ I mean, ‘We've agreed this is where we need to go. Now let's talk about what 
it's going to take to get us there for the children. And characteristically administrators are not viewed 
as people that do that . . . Administrators are viewed as people that tell you what to do, and you either 
like or you don't like it but, but you don't have a whole lot of voice in it ... I personally have to 
remember that there are good ideas out there that aren't mine, because for years administrators were 
expected to come up with the best ideas and the good way of doing it and that's what leadership was, 
and that's not what leadership is. Leadership, I have discovered, is trying to get everybody headed 
north at the same time.” 

Dr. Baldwin elaborated on the ways Quest had changed her approach to leadership. She 
reported, “I have learned to step back and to listen to the faculty rather than press my ideas and to 
not . . .be put on the offensive or the defensive when there is disagreement. And I have learned that 
that's an okay thing. Yeah, I think I've always felt before that everybody always had to agree and 
always had to be in step with everything in order to have a good working faculty, and I found out 
. . . that's not the case. And, that it's not my mission any longer to try to convince everybody to think 
alike. As a matter of fact it's much healthier and more productive to have people thinking differently 
because then what you get is this wonderful divergence of ideas and when you can finally marshal 
all of that and bring it together, then the solutions that you arrive at are really rich and unique.” 

Participants mentioned three times that they had become more reflective as a result of their 
involvement in Quest. For one Quest team member, this meant, “If s that kind of thing, when, if we're 
going to add that [program] what do we take away, and that's part of what this whole business is that 
I ponder about. I mean you don't go into things blindly any more. I don't simply accept them at face 
value ... I try to spend time thinking about the ramifications and what it's going to mean.” Another 
reported a sense of responsibility to think about and share information received at Quest events, 
although she also felt ambivalent about the depth of her efforts: “But that kind of thing has been very 
challenging ... I almost feel . . . just this weight of I need to share this. And so Quest has made me 
very reflective and very responsible, although I haven't, you know, necessarily felt that I've been 
affected enough in sharing the parent perspective.” 

Indicated twice was the way in which Quest had offered validation for participants’ 
perspectives on education and change. Such validation then led participants to feel more confident 
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about their views and philosophical commitments. According to one team member, “One of the 
things that Quest has done for me is to validate my perceptions about school leadership and my 

philosophy about how schools should operate and, and what their focus should For a long time 

I have felt as though I march to a different drummer ... in this school district for a long time .... 
And so too have other people from other places saying, ‘You're right, and this is, this is what ought 
to be done,’ has been very, very reinforcing and reassuring.” In addition, “I'm a whole lot more in 
perspective about those things than I used to be and I think, I think part of it comes from the 
assurance that that's an okay place to be. I think I shared in one of the rallies, education is the 
process of going from cocksure ignorance to thoughtful uncertainty, and I'm somewhere on the 
continuum. I hope [I’m] moving toward the thoughtful uncertainty, but I think I probably came in 
at the cocksure ignorance one, thinking I knew this is how you do things. Uh, so I think, I think 
Quest has done that for me.” 

Mentioned in two instances was the usefulness of Quest presentations on the implications 
of brain-based research for teaching and learning. Three other comments praised aspects of Quest, 
including the professionalism of project staff and the opportunities that Quest events provided for 
reflection on practice. One such respondent reported appreciating “the latest research that they 
[project staff] could give us in just that little bite sized pieces along the way and then you think about 
it and you hear it again at the next rally and that really does, it helps me a lot.” 

The five remaining comments were unique. One team member reported feeling more able 
and entitled to participate in the educational process at Tinder: “I think that what Quest has done 
for me it has empowered me to become an active participant in the education as it relates to school, 
and I think that's had, that's been a very positive impact of Quest.” Another suggested that Quest had 
assisted her in becoming more flexible, noting, “I think it's made me more flexible ... I think a result 
of that is due to AEL and Quest, that, that I'm more apt to be flexible . . . I'll be flexible and try this 
or anything that's going to help ... my students.” A parent participant indicated that involvement in 
the project enabled her to communicate more effectively with clients: “I think the information that 
I've gotten from Quest has allowed me to communicate to the students I have (because I teach on a 
part time basis), [it] has allowed me to communicate with them probably more on a one-on-one basis. 
And I've used some of things that I've learned in Quest, some things on brain-based learning ... and 
that has really helped me. And I think too, uh, when I produce a television program for the education 
department, I understand a lot better what's going on there and some of the language they use and 
the ideas and where they're coming from.” 

Finally, one team member reported designing personal professional development around an 
area discussed at Quest events: “I have professional development I'm working on this summer. One 
of my professional development days will have to do with technology and trying to enhance my 
teaching and the students learning through the, you know, the Internet. I have used the Internet for 
research this year as a result of what I learned, was learning and watching being done at Quest.” 

In addition, two respondents discussed the personal impact implementation of student-led 
conferences had made. A parent noted that listening to her child’s presentation had “allowed me to 
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understand what my [child] goes through on a day-to-day basis . . . You know, we’ve gone through 
parent conferences, you know, for the last four years, and you get a lot of information out of those 
from the teacher. But you don’t really hear much from the child. And with the student-led 
conferences done by Mr. Cook my [child] has had to speak up for himself for the first time in five 
years.” 



Another Quest team member described how student-led conferences had been a source of 
professional development: “The student-led conferences have just been really awesome. I did, 

I had a graduate class at [a local university], and for my master’s ... I did my whole research in that 
class based upon those and so it was great to have, have a lot of background information so that 
when I implemented in my classroom, I felt I was preparing my students better than I could have 
otherwise.” 

Two respondents also noted ways in which Tinder’s focus on enabling SMART learners had 
affected them individually. For one interviewee, this focus led her to better understand one of her 
students. She reported, “I think it also forces teachers to look at the SMART ways and what each 
child has that might be different. I'm thinking about one of my little boys that, you know, is not able 
to do paper pencil activities and ... I know in the past you know that that was your only measure. 
You would just have chalked this kid up, and yet when you give him the same thing orally you know 
that he has learned and he has obtained that information and that he is a very SMART learner. But 
he just leams in a different way.” 

One teacher discussed having personally designed a weekly reflection for students about the 
meaning of SMART learning. It was recounted as follows: “We have really put that into practice. 
The weekly reflection that I have shown with ... I think often in education, I, we, give students 
things and expect them to understand. But I, we, really had to break apart SMART. And still I have 
fifth graders at this school — I think you know I have a lot of really above average intelligence — and 
so I had, we had to really, really break down those parts and understand what successful meant and 
motivated and so forth, but that, that was important.” 

Impact of Quest on the School 

General Impact of Quest 

Tinder staff reported the impact Quest had made upon the whole school. Fourteen such 
comments were made, falling into one of seven categories of themes. Mentioned five times was the 
helpfulness to faculty of information and research presented by Quest staff. “[Quest staff] have time 
to read them and tell us the important things. And then if we want to know more, we can go read the 
book,” said one respondent. Another reported, “I think for, at least as far as the teachers are 
concerned, they're bombarded daily with new information. Y ou know, there's always a new magazine, 
there's an article ... I think that what Quest has done is take that bombardment and taken that 
information and put it in . . . useable areas, or in useable language, or in groupings or, you know, the 
things that are useful on a day-to-day basis ... It really has allowed them to take the information that 
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they wouldn't, couldn't, normally get to see, read . . . Well, what Quest has done is not only have you 
given us the news bulletins, but you've given us, you know, the information that goes with it. And 
you can pick and choose what you, what you want and what will work within the framework you 
already have.” 

Two respondents cited a list of needs assessment and self-analysis techniques of which they 
had learned from Quest and used at Tinder. Two others cited the Data on Display strategy as having 
been a useful way to work collaboratively. One described how this technique was used at Tinder: 
“The one thing that stands out in my mind is the dot system that, I don't know what you all call it, 
but, like, if there's something that has to be decided, instead of us all just raising our hand and saying 
what we think and then have to decide what all of us thought, we had the questions and or the list 
or whatever on the papers. We go in and put a dot on the ones we think is most important, and the 
ones that has the most dots is obviously ... a picture in our mind of what we think is the most 
important in our school. And I think that's the one thing that our faculty has used several times to 
solve a problem on what's the most important thing in teaching math or which is more important 
math or science or what do we need for . . . our consolidation plan or for choosing PD for our 
school.” 



Efforts to include parents more meaningfully in the life of the school resulted in increased 
parent involvement, according to two interviewees. One such respondent observed that alterations 
to the parking lot made her feel more welcome at Tinder. She said, “As I pulled into the parking lot 
just this afternoon I saw that . . . one of the changes that we've had in the school as a result — and I 
know it's a result of Quest because it happened right after one of our Quest rallies — is the parking 
spaces, parent parking spaces, were put closer to the door. Before it was the principal and the 
secretary and the administrative staff, and then the teachers and the parents could park at the end of 
the parking lot, which I didn't necessarily view as negative. And it wasn't until today that when I 
pulled up and parked right here next to the door I realized, wow, this is something that has been done 
to make us feel that we're important and that we're special . . . today I thought these are little things 
that are done in order to build our trust, to be able to build that community, that relationship.” 

The remaining three replies were unique. One respondent reported that Quest had encouraged 
team members to engage in reflection as a group, elaborating, “I think that is one of the big impacts 
that Quest has had. It's made us stop and think. And it's made us stop and think as a group, as Quest 
members and as individuals, you know, ‘What is my impact on the school? What can I do? What am 
I capable of?’” 

Another interviewee suggested that Quest had reinforced the significance of professional 
development, which had enhanced Tinder staffs professional lives. As she put this, “I think in a 
broad sense it's helped me to understand the importance of professional growth, not just for myself, 
but for the faculty, and that the ideas and the things that we do here don't have to come from, from 
the principal. They need to come from other places and other people. And having a team involved 
in Quest and what we've learned uh, has had an impact. And I have, you know, my teachers and the 
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parents that have been involved in Quest are different people as a result, and I think Quest has given 
them the courage and the knowledge to do things differently than their colleagues.” 

One respondent described the ways in which information provided at Quest events about the 
applications of findings from brain-based research has been incorporated into the school. “I know 
certain teachers are more conscious of these students, making sure that they're hydrated and that they 
have protein, you know. We always make sure they have healthy snacks and things like that . . . what 
we've learned in brain-based research.” 

Use of the SMART Learner Theme 

Interviewees also offered 26 comments constituting seven categories of themes about the use 
of the SMART learner theme throughout the school. Eighteen comments simply named or described 
the various ways SMART had been woven into the school culture. Three respondents each reported 
that the SMART theme had been displayed in myriad ways on bulletin boards throughout the school, 
a parent wrote a monthly column in the school newsletter based on the theme, and several workshops 
had been conducted by parents for parents on ways they could incorporate insights from the theme 
into their parenting. Mentioned twice each were the inclusion of SMART as a theme throughout the 
yearbook, the renaming of the theme to become more inclusive (“We are Totally SMART!”), the 
SMART creed recited daily, and SMART stationary. One respondent also reported that Totally 
SMART t-shirts had been created and distributed at the school. 

However, according to two interviewees, the adoption of the theme took some time. As one 
phrased it, “I think that, that probably the children at first did not understand, and it's difficult for the 
parents to understand what SMART is. And it's taken a while for that to really take hold for the kids 
to understand the individual parts of SMART and, and to understand what that means as a whole.” 
In addition, one other respondent reported that the SMART learner creed recited daily was often read 
too quickly over the intercom, not providing students enough time to reflect on its meaning. 

Three comments, on the other hand, suggested that the concept now permeates the school. 
According to one interviewee, “From listening to Nadia this morning when she was mentioning how 
her child, you know, when he was sharing something with her in the grocery store and then he says, 
‘Well, I'm being thoughtful, aren't I?’ You know, I think that it permeates ... or trickles down, 
whatever you want to call it, you know, from the teacher to the student, you know, to the parent . . 

. I think that the parents and the teachers and the students will be more on the same, same wave 
length and be able to understand more what being SMART is ... I think that it will just become a 
part of us — I mean, it has to an extent already in one year’s time — but I think four years down the 
road it will just be, we won't remember what it was like without it, maybe.” 

Another respondent explained, “I think . . . that's a common theme, that cuts across all levels 
of instruction, kindergarten to sixth grade, across all adults regardless of their role or mission in the 
school, and that if you ask someone about the school, ‘What's it all about?’ the children might say 
‘SMART learners,’ or they might say, ‘Totally SMART.’ It gives an identity, if you will, and I think 
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it also, it also gives a framework for doing things in the classroom . . . we're doing this because we're 
responsible, we're doing this because we're autonomous, we're doing this because we're motivated. 
It gives us, it gives us a framework for looking at each of those things too . . . what is motivation all 
about? I mean that's, that's a study in and of itself.” 

Two respondents discussed ways they helped students consider the meaning of the acronym. 
One reported that classes set aside time to discuss what each word in SMART meant. Another 
described a weekly reflection sheet a teacher had developed for students. Items asked students to 
write about ways in which they had been successful, motivated, autonomous, responsible, and 
thoughtful during the week. 

The remaining two replies were idiosyncratic. One respondent noted that the principal often 
used the acronym during interactions with students. And one reported conducting graduate research 
at Tinder to explore whether a decline in assignment alerts would result from the schoolwide focus 
on SMART. However, at the time of the interview, data were not yet available for her to assess this. 

Use of Student-Led Conferences 

Sixteen comments fell into six categories about the ways student-led conferences had affected 
Tinder. Respondents made four general positive observations about student-led conferences. For 
instance, one such interviewee reported, “Going to these Quest rallies has really been a good avenue 
for me to learn new things, because without some of those I don't know, you know, I probably would 
still be having the parent teacher conferences that I never did like any way.” Another shared that 
student-led conferences had been a revelation to her: “And the student-led conferences has been a 
tremendous thing that has come out of Quest. I had never even heard of such a thing until Quest.” 

Four comments suggested that the adoption of the SMART learner component of the Quest 
framework had inspired Tinder’s use of student-led conferences. Said an interviewee, “And with the 
SMART, I think student-led conferences come directly out of that, especially in being autonomous 
for everything they do, because they're in charge of the conference, and being responsible.” Another 
respondent, apparently not directly involved in conducting student-led conferences, noted that they 
appeared to reinforce the ‘T’ in SMART: “I think the T is [part of it] too. I think one of the things 
you all were doing with the student-led conferences that was so valuable is that whole business of 
reflection.” 

Two Tinder staff reported that those teachers piloting student-led conferences would use their 
experiences to train other teachers in the building. Similarly, three interviewees shared the ways in 
which the innovation had been adapted to better meet their needs. One, for example, described a 
colleague’s use of both student-led and traditional parent teacher conferences. She said, “I mean, 
she's talked about next year maybe not doing the student-led conferences the first quarter, [she’ll 
just] go ahead and do the parent teacher conferences because there is such a difference between 
primary and intermediate that the parents needed some feedback maybe from the teacher. And then 
she had talked about doing the parent one, the parent teacher first quarter and then doing the student- 
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led second/third quarter.” Another described her revisions of the process: “Well, the first time that 
we had them I think that some of the parents were a little nervous and the kids were nervous . . . 
We'd prepared the portfolios, and we did the reflecting, reflect and justify on the pieces that they 
chose and then graphed their grades so they could see . . . That was an excellent visual. And they 
were able to set goals and I had, and after each set of conferences I would revise, I would go back 
and reflect and see how you know these things worked . . . perhaps revise their goal setting sheets 
and try to make it, try to facilitate it a little bit more, and help them to focus and instead of just 
saying, ‘What goals are you going to work on?’ ‘Well, what goals are you going to work on, and how 
are you going to do, what are you going to do in order to achieve those goals?’” 

Parent attendance at conferences had improved, according to two Tinder staff members. One 
shared her sense of this: “I'm not aware of any parents that didn't come. And when we have this, the 
teacher conferences, we maybe will have four or five that don't show up from the whole classroom 
but I'm thinking that they had a really high attendance rate.” 

Preparing students for leading conferences with their parents was a learning opportunity in 
and of itself, indicated one respondent, reporting a colleague’s experience: “I think that one of the 
teachers in the primary had said that she was, she was more aware of the terminology that she used 
because she wanted her students to be able to convey that to her parents. And you know, how could 
they if she didn't use the word proofread and edit and those kind of things that they probably 
wouldn't with their parent either? So she said, ‘It just really forced me to make sure that they were 
going to have the information they needed to present that to their parents in a meaningful way.’” 

Use of Math Action Research 

During the 1999-2000 school year, Tinder staff noticed a decline in students’ math 
computation scores on the spring administration of the CAT5, the state-mandated standardized 
achievement test. Puzzled by the decrease in scores and the simultaneous gains made with regard to 
math concepts, several staff began to investigate. As reported in the faculty focus group, they 
administrated questionnaires to students and teachers in an effort to isolate the factors effecting the 
scores (see Appendix for the full transcript of teachers’ account of this venture). However, the data 
from these identified too many variables for teachers to address, so staff requested assistance from 
Quest staff. 

Quest staff in turn referred the concerned teachers to staff at the Region IV Comprehensive 
Center located at AEL, who led Tinder in an action research project focusing on math computation. 
Comprehensive Center staff interviewed a sample of Tinder teachers about their math instruction, 
examined the student data, and analyzed the questionnaires completed by school faculty. In addition, 
Tinder staff participated in a half-day professional development session on various instructional 
techniques that could be used to enhance students’ computational skills. 




41 



36 



Teachers’ analyses of test data also revealed several specific areas of concern, including 
computation with fractions, decimals, and place value. As a result, teachers now often begin their 
lessons with a review of these areas, engaging students in solving computation problems. 

Teachers also discovered that the issues with declining computation scores were not endemic 
to particular teachers or classes. Rather, it appeared to be a schoolwide challenge. 

At the time of the case study site visit, however, test data were not yet available for teachers 
to assess the results of their efforts. Nonetheless, they reported seeing some changes in students’ 
performance. Said one teacher, for example, “Now myself, since that [professional development] 
meeting, everyday I start my math class . . . review problems on the board . . . and I do see it, they’re 
getting better at them. When we first [started], some of them were only getting two right. And now 
most of the class are getting all five right. And I vary what I ask. I ask geometry questions, and each 
of the five are different. I don’t, like, have five addition . . . subtraction, and multiplication.” 

Student Outcomes 

Outcomes of Student-Led Conferences 

Interestingly, interviewees made relatively few comments about student outcomes that they 
could attribute to participation in Quest. Respondents made nine comments, falling into four 
categories, about student outcomes of involvement in student-led conferences. Three brief comments 
indicated that students simply enjoyed the chance to present their work. Students felt more 
ownership for their school work as a result of leading conferences with their parents, reported two 
interviewees. For instance, one Quest team member suggested, “It makes sense. If the child is saying, 
‘You know, I really need to work on this,’ as opposed to the teacher saying to the parent that your 
child really needs to work on this ... It makes sense that they would be more [involved].” 

Two team members indicated that students engaged in more reflection about their work due 
to their participation in student-led conferences. Asked about student outcomes from the 
conferences, one individual replied, “I don't know, unless it's more of an ability to reflect on what 
we've done. Because I know they did quite a bit of reflection work, so that's something that probably 
we don't slow down and take time to do often, and I don't know that they do [it] often. But maybe 
just right after that conference they can reflect on their work from the quarter.” 

Two respondents commented that students now developed improvement plans for 
themselves, one of whom additionally reported that students were more responsible for their work 
as a result of the student-led conferences: “They keep up with their work better, you know. I have 
them keep their things in a 3 -ring binder ... I think that they're more responsible to keep up with 
their work and it's given them an area to organize their things. I've been very pleased with it. I think 
that, I've been, it's helped me, which has enabled them to be better goal setters, because usually, you 
know, you set goals at the beginning of the year and well, you don't look at them, you know, until 
the end of the year. You don't go back and look at them like you should, and I've been guilty of that. 
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And this has, you know, we've reviewed them frequently and they have a copy in their folder of their, 
of their goals and check it . . . I've been very excited about that.” 

SMART Learner Outcomes 

Interviewees discussed student outcomes from the school’s adoption of the SMART learner 
theme, making five comments falling into four categories of themes. Two respondents reported that 
students appeared more responsible. One, for instance, said, “I think that probably students are not 
as afraid to take responsibility for things they've done, both good and bad.” 

Another respondent indicated that students seemed increasingly thoughtful as a result of 
Tinder’s focus on the SMART learner component of the Quest framework. She reported, “I think 
that they are probably a little more thoughtful in the things that they say and what they do. Uh, I think 
in some ways they probably maybe look at the entire education process or their entire class a little 
differently than they did before, you know. How do these things impact other people? Most kids are 
really self-centered, and it's really difficult to get them to look outside themselves. And I think in 
some ways this has gotten them to look outside themselves at other people in the class. And at least 
I can see some of that going on in my son’s class.” 

One respondent each reported that students were more confident in their ability to learn new 
material and used the words contained in the SMART acronym. 

Benefits of Network Participation 

Interviewees reported a variety of benefits derived from the Quest network of schools, 
making 24 comments falling into 1 1 categories of themes. Mentioned six times was the way in which 
the network facilitated the exchange of ideas, coupled with the inspiration to implement the most 
useful of these. For instance, one such respondent said, “Mansfield and some of the other schools 
that we've talked to give us a direction; they give us ideas; they give us feedback. I mean we look 
at some of the smaller schools that have fewer resources and have different populations] than we 
have, and I mean, they're a world apart from where we are, and yet they achieve so much and they 
do indeed have a sense of community inside their school. I mean we envy, you know, some of the 
things they've been able to achieve.” Reported another, “And I think that the ideas that those schools 
have brought into Tinder and just the examples that they have shown us, you know, what is possible, 
has really given us kind of a foundation, if not something to head toward, a real goal that we can do. 
If somebody else can do this, so can we.” 

Similarly, five interviewees noted that the network enabled participants to share ideas and 
resources. One such respondent reported, “I've talked to . . . Nathan, and I've talked to Helen and 
Mrs. Vincent, and getting to know them as individuals always helps, and you're not afraid to offer 
suggestions or offer to help. If I know someone on a personal basis, you know, I'm willing to say, 
‘Hey, anything I can do please let me know.’ I'm willing to put myself out there.” For another 
participant, the various means of communication were helpful. She said, “The e-mail business is 
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wonderful and the listserv too. I mean, I don't always respond when those things come out, but I 
know that there are people out there thinking and sharing ideas and using it as a sounding board, and 
that's a good thing too. And sometimes I, I share some thoughts but sometimes I, I just read what's 
out there too. But I've taken lists of things off there and, and used some of that information, followed 
up on it, went to a website that was, that was listed as a good resource, or looked at grant possibilities 
because they've been listed, those kinds of things. And then of course . . . from the network is 
resources that people are using in their schools that we hadn't heard of that we know, we have, have 
books that we purchased as a result, or CD's.” 

Two respondents reported that a benefit of the network had been the provision of information 
about a variety of school improvement strategies. Schools could then choose from them the most 
appropriate strategy to meet their specific needs. As one interviewee put it, “It's certainly given us 
a lot of things to think about. And some things that are working for other schools are not necessarily 
going to work for us, and that's good. I mean, we can, we can pick and choose from the things we've 
learned from the network of schools.” 

Another two respondents reported that the network allowed them to see their schools from 
new perspectives. For instance, one Quest team member shared, “I think that ... the network . . . 
provided some recognition of the things that we're doing at this school that to us seem common place 
and other people find to be very extraordinary. I mean, I think, that was very supportive. It's, you 
know, [as if network participants are saying], ‘Don't give up what you're doing. That's a special kind 
of thing you're doing.” You know, I hadn't thought about it that way. So it's given us, it's given us 
that.” 



Two interviewees expressed their admiration for other network participants. As one phrased 
this sentiment, “I have such admiration, not only friendship, but admiration, for the people that I 
have met in the network. The Saint Margaret people impressed me so. They've got their own set of 
constraints, you know, and the things that they're doing within their mission is really impressive. 
So there hasn't been anyone that I haven't learned from in the network.” 

A visit from another network school was mentioned in two instances. Reported one faculty 
member, “[A fourth grade teacher from Bowman] came and spent some time with me and we shared 
some things and I shared with her the things that I was doing you know as far as the reflecting and 
she took copies of different things and my parent letter that I do each Friday and the scoring rubric 
and you know just everything that's involved and what I give the children for that to help them with 
their reflecting and communicating with parents and we just, we shared some things.” 

The remaining comments were idiosyncratic. One respondent expressed appreciation for the 
opportunity to interact with students participating in the Quest high school network. Another 
reported that the network helped support the reflection necessary for continuous improvement. 
Relationships within her Quest team had been strengthened as a result of participation in the 
network, noted a team member. 
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One respondent reported that although Quest staff used a variety of “buzz words ... I just 
had to get through the language to get to the meat of the process.” Later she added that she was 
pleased to be included in Quest as a parent; nonetheless, she felt that the project was intended as a 
“professional type of networking organization.” 

Three additional comments were made by interviewees about the value of networking with 
schools from states other than Kentucky. Two reported that meeting participants from other states 
allowed them to look beyond their own particular education contexts. As one put it, “I think that in 
Kentucky we have a mind set — and sometimes it's good and sometimes it's not good — we think 
we're on the top of the heap as far as education reform. And in some ways we probably are, and in 
other ways we're really behind, and they don't recognize the fact because they're, they're constantly 
looking at themselves and comparing themselves to other schools in the area or other schools in the 
state. Well, they're all working with the same materials. When you look at, you know, at a school 
that's in Georgia, or one that's in Tennessee, or one that's in West Virginia, they have a whole other 
set of rules they have to live by, and you know, resources that they can use, and you can look at that 
and draw inspiration from it.” 

Differently, another respondent felt that networking across states revealed the similarities 
between schools. “The value [of networking with schools outside of Kentucky] is that we're all in 
the business of helping children learn, and we can all learn from one another because none of us has 
the comer on all the right ideas. And although situations may differ, we can still leam from one 
another.” 




45 



40 



CONCLUSIONS 

The data collected suggest several conclusions about the impact participation in Quest had 
upon Tinder Elementary. 

Although the school was highly involved in the project in many respects, including 
attendance at every Quest gathering, there were some limitations to their participation. For example, 
only one of the two school administrators attended Quest events; the school hosted a visit from 
another Quest school, but did not participate more fully by assisting a sister school implement an 
effective practice; and project team members did not believe faculty and staff experienced a dramatic 
change in their view of school improvement. 

Tinder undertook a few initiatives which they attributed to participation in Quest. The school 
adopted the SMART learner theme as a focal point, using the phrase in yearbooks, newsletters, 
parent workshops, billboards, and t-shirts. Several teachers piloted the use of student-led conferences 
in place of traditional parent-teacher conferences with success. Staff also undertook an action 
research project to examine the impact of changed instructional practices on their students’ math 
computation performance. 

Quest team members clearly thought that Tinder approximated well many of the Quest 
framework components. However, shared leadership appeared to be a concern both to Quest team 
members and the principal herself. 

Tinder’s total score on the Professional Learning Community instrument decreased slightly 
between the 1 998 and 2000 administrations. However, the decline was not statistically significant, 
suggesting that it was perhaps due to chance and not to a deterioration of Tinder’s sense of 
constituting a learning community. Nonetheless, Quest did not appear to have an effect on the 
school’s development in this regard. 

The project also had little impact on student achievement at Tinder, although improvement 
trends in grades three through five were apparent, and a few statistically significant improvements 
were located. These, however, were not substantial, and therefore not of much practical meaning. 
Such findings are not surprising, given Tinder’s use of Quest for purposes other than explicitly 
targeting achievement scores. 

On the other hand, student TCS/2 total, nonverbal, and verbal scores were consistently higher 
in 1 998 and 1 999 than in 1 996, when Tinder began its involvement in Quest. Moreover, these gains 
were statistically significant. Similar statistically significant differences were located within grade 
levels, although none were found in the fifth and sixth grades. Nonetheless, these findings should 
be interpreted with caution, as the higher scores may be an artifact of the slightly older sample of 
students in 1999. 
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Although statistical significance indicates that findings are not likely due to chance or 
sampling errors, the small effect sizes for TCS/2 gains suggest that the impact Quest may have made 
upon scores was nonetheless small. This is not surprising given the many factors influencing the life 
of a school. 

Interviewees were quite clear that Quest had made important contributions to their personal 
and professional growth. From learning more about the educational process to becoming more 
reflective to focusing on sharing leadership, Tinder school community members indicated that 
participation in the project had been significant at the personal level. 

Quest provided Tinder staff useful information gleaned from a variety of sources in a non- 
prescriptive manner, allowing the school to choose which resources were most aligned with its goals 
and concerns. Due to participation in Quest, Tinder became more attentive to nurturing meaningful 
parent involvement, for instance. These were important contributions to the school’s development. 

The incorporation of the SMART learner theme into the school culture appears to have 
breadth and depth. In other words, the theme is displayed and referred to in a variety of settings and 
media, and it is reported to have influenced specific decisions, such as to pilot student-led 
conferences and parent workshops. Students seem to be more thoughtful and responsible, according 
to some Tinder staff, as a result of the school’s focus on enabling SMART learners. 

Student-led conferences were piloted successfully, and teachers plan to train other teachers 
on the faculty interested in conducting them as well. Teachers believe that students have benefitted 
from the innovation, becoming increasingly reflective about and responsible for their work. 

Although math computation scores are not yet available to assess Tinder’s action research 
efforts, teachers interviewed thought that the process had been useful. Several believed that 
improvements in students’ computation skills were already apparent. 

The Quest network facilitated the exchange of information and resources among participants, 
and nurtured a variety of relationships as well. These served as a source of inspiration and knowledge 
as Tinder staff participated in the project and undertook improvement efforts. 

In sum, it appears that Quest made uneven impact on Tinder Elementary during the few years 
of the school’s participation. For instance, while TCS/2 scores rose over the course of the school’s 
involvement in the project, Professional Learning Community scores dropped slightly. Although 
student-led conferences were used successfully by several teachers, most had not implemented the 
innovation. And interviewees reported relatively few student outcomes which they believed they 
could attribute to Quest. Student achievement data also indicate few statistically significant 
improvements over the course of the school’s participation in the project. Nevertheless, interviewees 
perceived that Quest had made significant, meaningful contributions to the school’s development 
and to their own personal and professional growth. 
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It is the impact of the project on Tinder faculty and parents that holds particular promise for 
continuous improvement at the school. In other words, their understandings of school change may 
facilitate new initiatives or expanded use of strategies learned during the school’s involvement in 
Quest, the outcomes from which may not be apparent until after the end of the project. Having 
experienced significant professional growth and having enacted a variety of Quest techniques, Tinder 
is well-equipped to continue on its trajectory of ongoing improvement. 
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Quest for Quality Learning Communities 

A Program for Continuous School improvement 



School improvement is challenging work; to be 
effective, it must be continuous. Improvement is 
not a single act or program; it is a process of al- 
ways wanting to learn more about how better to 
help all students achieve at higher levels. Improve- 
ment is visionary; it involves risk-taking, uncer- 
tainty, and a rejection of “doing what we’ve 
always done.” Most of all, improve- ^ ^ , 

ment requires more than indi- ^ 

vidual effort: it is a collabo- 
rative endeavor that en 
gages and responds to 
the diverse voices 
within an entire * 
community. 

Teams from 20 
schools in a four- 
state region now 
collaborate with 
staff from the Ap- 
palachia Educa 
tional Laboratory 
(AEL) to study and 
learn together in the 
Quest project, and each 
school takes a slightly different 
path. For example, one school tar- 
gets increased parent involvement; an- 
other hopes to raise the level of student thinking 
through teachers’ working together and coaching 
one another; a high school improves teaching by 
listening to what students say about how they 
learn best; other schools focus on specific curricu- 
lum areas such as writing or science education. 

The Quest framework unifies their thinking about 
school improvement. These core values offer a 
blueprint for continuous progress: ongoing ques- 
tioning of practice, high expectations for all, in- 



dividual responsibility for better performance, col- 
legial sharing and support, and thoughtful reflec- 
tion on practice. 

Stemming from these values is a clearly defined 
vision of student excellence that is shared by all 
members of the school community. A strong 
learning culture encourages both stu- 
^ ^ ^ dents and teachers to choose con- 

tinuous improvement as a way 
of life in their school. Mem- 
\ bers of the school com- 
munity connect to one 
another through a 
shared commitment 
to improved learn- 
ing conditions for 
all. Shared leader- 




ship encourages and 
enables everyone to 
assume responsibility 
for making a positive 
/ impact on the school 
community. Shared goals 
for student learning moti- 
vate individuals to improve 
their performance and help focus 
the energies of the entire community. 

The collection, analysis, and use of student as- 
sessment data sustains continuous improvement, 
providing a measure of the effectiveness of the 
community’s efforts. SMART learners are Suc- 
cessful, Motivated, Autonomous, Responsible, 
and Thoughtful. Fully equipped to become life- 
long learners, they are ready for life and work in 
the 21st century. In short, continuous improve- 
ment spawns the energy and excitement neces- 
sary to transform a collection of individuals into 
a true learning community. 




Goals of the Quest Project 



1 . Connect with colleagues. By serving on a Quest leadership team, participants connect with others on their school 
team, forming bonds that enhance working relationships. In addition, Quest teams connect with teams from 
odier schools, districts, and states, allowing everyone to learn from others’ experiences. A Iistserv, inquiry@ael.org, 
facilitates connections across the network. 

2. Create a learning community. Teams become part of the Quest net- 
work learning community with the expectation of recreating this ex- 
perience in their own community, 

3. Connect vAxh. concepts and stories related to continuous school im- 
provement. At Quest rallies, the Quest framework is a source of study, 
dialogue, and sharing among teams. 

4. Create personal and shared meaning. The Quest network places a 
high value on processes such as reflection and dialogue, which lead to 
deeper understandings of continuous improvement. 

5. Commit to continue learning with this community. Quest schools 
have made a three-year commitment to study and learn together, with 
a focus on improving student achievement. 

6. Commit to continue the Quest back home. The “rubber hits the road” at schools, not at Quest events. AEL helps 
school teams take their learnings home and apply them for the benefit of students. Site visits, called Co- Ventures 
in Learning, provide opportunities for AEL staff to visit each school, in order to better understand the context of 
that schools efforts, and tailor assistance to the school’s needs. 



What Is a learning community? 
“Learning communities are essen- 
tially communities of Inquirers . . . 
sustained by a continued commit- 
ment to share this journey of 
exploration with one another on 
matters people care deeply about” 
(Ryan, 1995). 



Peter Senge et al. (1 994) write that a 
learning organization “Is a place 
where people continually expand 
their capacity to create the results 
they truly desire, . . . and where 
people are continually learning how 
to learn together.” 



Hie Quest project hopes to achieve results at three different levels: 

For individuals, sharing leadership on a Quest team leads to more reflective practice and renewed under 
standing of the concepts that support continuous improvement. 



For schools, Quest will provide motivation and support for ongoing and/or new school-based initiatives to 
improve teaching and learning. 



For the Quest network of schools, our collaborative learning and research will yield stories, insights, 
processes, and products — all of which will be helpful to the broader educational community. 
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Quest Co-Venture 2 Interview Protocol 



Instructions: We are interested in your perceptions regarding the difference Quest has made for you 
' personally and for your school. Please respond to the focusing questions honestly and openly. 

There are no right answers, and we are sincerely interested in your personal assessment and 
reaction. Also, please be assured that your name will not be associated with any of your 
comments. We are committed to protecting your confidentiality and anonymity. 

1. Personal/Professional: What impact has your involvement with Quest had 
upon your personal and professional growth and development? 



If the respondent does not talk about the following areas of impact, the interviewer should probe for more 
t information using the prompts below. 

(a) In what areas have you increased your knowledge and skills as a result of Quest 
participation? 



(b) What attitudes or beliefs have been challenged and/or modified through involvement with 
Quest? 



(c) What questions have you been prompted to investigate? 



(d) What personal behaviors have you changed or attempted to modify as a result of your 
involvement in Quest? 



(e) What meaningful relationships have you developed? 



2. School reform/improvement: In what ways has participation in Quest contributed to your 

school's journey of continuous improvement? 



(a) Please describe specifically the results or outcomes for students and adults that you believe 
attributable to Quest. 



(b) Think about the Quest framework and its six constructs. In which of these areas has your 

school become more focused? 

Culture for Learning 
Sharing Leadership 
Community of Learners 
SMART Learners 

Assessing and Demonstrating Learning 
Sharing Goals for Learning 

(c) What is different now as a result of your school’s focus on this component? 



3. Value of Network: To what extent and in what ways has the Quest network supported the 
individual learning and school improvement you described above? 

Additional prompts for further information below. 

(a) What is the value of networking with others schools as we have done in Quest? 



(b) Describe the value of relating to schools outside of your own state. 



APPENDIX C: 

Faculty Focus Group Interview Protocol 
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Quest Co-Venture 2 
Faculty Focus Group Protocol 

Thank you very much for participating today in this focus group interview. We’re interested in 
learning more about what school improvement efforts you’ve undertaken here since your 
school’s involvement in the Quest network. 

Let me describe a few guidelines before we begin. First, we will be recording today’s 

conversation. However, let me assure you that your name will not be associated with any 
comment you make. We will have the tape transcribed, but you will never be identified 
personally. This is to protect your confidentiality and anonymity. Second, the purpose of a 
focus group is to get everyone’s candid viewpoint. No one’s answers are right or wrong, 
so please respect everyone’s opinion. And, finally, it is important that everyone has an 
opportunity to express their opinions concerning each question. It is my job to ensure that 
everyone has that opportunity. With these guidelines in mind, let’s begin! 

1. For those of you not directly involved in Quest, what is your understanding of the project? 



2. What school improvement efforts have you undertaken here as a result of your school’s 
participation in Quest? 



3. How successful have these been? For what reasons? 



4. What student results or outcomes have you seen as a result of the improvement projects you’ve 
undertaken? (Do you have data or stories supporting this that you might share with us?) 



5. What other results or outcomes have you seen as a result of these projects? (Do you have data 
or stories supporting this that you might share with us?) 



6. What is your favorite story about the projects you’ve participated in? 



APPENDIX D: 



Student Focus Group Protocol 



Co-Venture 2 Student Focus Group Protocol 



My name is , and I work at an organization called AEL. We have been working 

with your school to find out more about good teaching and learning. Today we’re going to ask 
you a few questions about being a student here. We are tape recording our conversation, but no 
one at your school will listen to the tape, and any comment you make will not be associated with 
your name. 

There are a few guidelines for this kind of discussion. First, we’re talking about opinions today, so no 
one’s answers are right or wrong. Please respect each other’s answers. And second, it is my job 
to make sure that everyone has a chance to speak. Please be considerate and take turns when 
answering. Any questions? 

Let’s begin. 

1. Remember an experience in school when you felt very successful as a learner. Please you 
describe that experience to us. 

Probes: 

What was it that made it successful? 

What did you do? 

What did the teacher do? 



2. Remember an experience in school when you did not feel successful as a learner. Please describe 
that experience to us. 

Probes: 

What was it that made it not very successful? 

What did you do? 

What did the teacher do? 



3. How do you learn best — or how do you like to learn? Try to elicit the types of instructional 

approaches that “work” for these students. 
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5 . 



Elementary. Is school easy or hard for you? For what reasons? High School. Do you feel 
challenged in school? Are there opportunities to be engaged in challenging courses? 



6. High school. What kind of schedule does this school operate on (traditional; 4x4 block; AB 
block; other). How well does this work? 



7. What are some words you would use to describe this school that capture “what is important 
around here”? 



8 . 



What else would you like to tell me about learning in this school? 



APPENDIX E: 

Reflective Assessment Questionnaire 
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Reflective Assessment for Quest Schools of Continuous Improvement 



Name: 



School: 



Directions: This instrument was designed to help you reflect upon your school’s development 
as a school of continuous improvement. For each of the six dimensions of the Quest framework, 
circle the number that best represents your school’s current position on the continuum. Then 
explain your rating and describe how change has occurred. Please be honest. We appreciate 
frank and open responses. 

Also, please be assured that your name will not be associated with any of your comments. We 
are committed to protecting your confidentiality and anonymity. 



Culture for Learning 



The paragraph below describes a school at “100” in the area of “culture for learning.” 
Where does your school fit on the continuum? 

Members of the school staff frequently reflect on how to improve the school for all 
students. They not only ask lots of questions-including “How can we do this 
better?”-but also they regularly try new ideas, with administrative support, and 
celebrate their successes as a community. Likewise, students are curious and show 
excitement for learning. 

100 90 80 70 60 50 40 30 20 10 0 



On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest made an impact? Give specific examples. 



T 

If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 



Community of Learners 



>- The paragraph below describes a school at “100” in the area of “community of learners.” 
Where does your school fit on the continuum? 

Members of the school community, especially students and teachers, feel connected to 
one another and to the school as an organization with a clear mission. Open and 
regular communication promote norms of trust and respect. The school is a center of 
learning for the entire community; parents and other community members are 
welcomed and valued. 

100 90 80 70 60 50 40 30 20 10 0 

>- On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



>- Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest made an impact? Give specific examples. 



T 

If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 
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Sharing Leadership for Learning 

The paragraph below describes a school at “100” in the area of “sharing leadership for 
learning.” Where does your school fit on the continuum? 

Teachers, parents, and students have a forum for input into decisions and have easy 
access to important information about the school. They know that they are listened to 
and that what they think and do makes a difference. School administrator(s) 
participate democratically with teachers. School leadership teams include students-or 
at least are guided by students’ perspectives-as they plan for school improvement. 

100 90 80 70 60 50 40 30 20 10 0 

On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest made an impact? Give specific examples. 



If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 



Shared Goals for Learning 



>■ The paragraph below describes a school at “ 1 00” in the area of “shared goals for learning.” 
Where does your school fit on the continuum? 

Goals for school improvement are specific, measurable, and identifiable by all 
segments of the school community. These goals are a major consideration in decision 
making about allocation of school resources. They affect decisions at both the 
classroom and school levels. 

100 90 80 70 60 50 40 30 20 10 0 

> On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



> Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest hade an impact? Give specific examples. 



▼ 

If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 
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Assessing and Demonstrating Learning 

>► The paragraph below describes a school at “ 1 00” in the area of “assessing and demonstrating 
learning.” Where does your school fit on the continuum? 

At the school level, multiple data sources are carefully studied and used in setting 
goals. Results of student achievement tests are disaggregated and are widely 
communicated and interpreted to the broader community. At the classroom level, 
teachers communicate clear expectations for student performance and use a variety of 
methods to assess progress. Students and teachers actively assess their own 
performance and time is provided for this reflection. 

100 90 80 70 60 50 40 30 20 10 0 

> On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



>- Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest hade an impact? Give specific examples. 



▼ 

If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 
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Enabling SMART Learners 

>- The paragraph below describes a school at “100” in the area of “enabling SMART learners.” 
Where does your school fit on the continuum? 

Throughout the school, students and teachers are actively engaged in meaningful work 
which they understand is connected with the real world and with their future. 

Students are aware of their own personal strengths in learning, they increasingly arc 
intrinsically motivated to learn, and accept responsibility for their own performance. 
Students and teachers arc aware that learning discrete facts is not nearly as important 
as is developing skills necessary' for lifelong learning in the complex world of the 21 st 
century'. 

100 90 80 70 60 50 40 30 20 10 0 

>- On what do you base your rating? Cite specific evidence/examples to support the above 
rating. 



>■ Has involvement in Quest for Quality Learning Communities influenced your school’s 
development in this area? 

□ No □ Yes 

If yes, in what ways has Quest hade an impact? Give specific examples. 



▼ 

If yes or no, what besides involvement in Quest have been the major factors influencing your 
school’s development in this area? 
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APPENDIX F: 



Professional Learning Community Instrument 
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School Professional Staff as Learning Community 
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School Name: >•' Your Role/Position 

Dale: 



Quest Schools of Continuous Improvement 
Innovation Configuration Checklist 



Directions: The eight items in this instrument represent the components associated with the 
Quest Network of Schools of Continuous Improvement. Beneath each component are 
alternative ways in which the components might be implemented in a school. For each item, 
circle the letter which best describes your perception of how Quest has been implemented in your 
school. 



1. School Leadership Team 

a. All three of the following are true of our school leadership team: (1) It is inclusive with 
administrator, teacher, parent, and (in the case of high schools) student membership; (2) It 
has been fairly stable over time; and (3) It has assumed active leadership in taking our 
quest back home to the broader school community. 

b. Two of the following are true of our school leadership team: (1) It is inclusive with 
administrator, teacher, parent, and (in the case of high schools) student membership; (2) It 
has been fairly stable over time; and (3) It has assumed active leadership in taking our 
quest back home to the broader school community. 

c. One of the following statements is true of our school leadership team: (1) It is inclusive 
with administrator, teacher, parent, and (in the case of high schools) student membership; 
(2) It has been fairly stable over time; and (3) It has assumed active leadership in taking 
our quest back home to the broader school community. 

d. Our school does not have a true leadership team. 



2. Administrative Support 

a. One or more of our school’s administrators are active members of the leadership team 
and extensively involved in Quest activities. 

b One or more of our school’s administrators are members of the leadership team and have 
been occasionally involved in Quest activities. 

c One or more of our school’s administrators have been involved in a few Quest activities 
and have been generally supportive of our team. 

d Our school administrators have not been involved in Quest activities and are only 
minimally supportive of our school’s involvement in Quest. 
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3. Participation in Network Events 

a Individuals from our school have participated in three (3) or more Network events during 
the past year including Rallies, Summer Symposia, and/or Scholar s CoIIoquia. 

b. Individuals from our school have participated in two (2) Network events during the past 
twelve months including Rallies, Summer Symposia, and/or Scholar s CoIIoquia. 

c Individuals from our school have participated in one (1) Network event during the past 
twelve months including Rallies, Summer Symposia, and/or Scholar s Colloquia. 

d. Individuals from our school have not participated in any Network events during the past 
year. 



4. Participation in Co-Ventures in Learning 

a. A wide cross-section (i.e., administrators, teachers, staff, parents, and students) of our 
school community participated in the Quest Co-Venture(s) in Learning. 

b. A limited number of our school community — primarily administrators, teachers and 
staff — participated in our Quest Co-Venture(s). 

c. Our school has not yet engaged in a Quest Co-Venture in Learning, but plans to do so 
during this school year. 

d. Our school has no plans to participate in a Quest Co-Venture. 



5. Involvement with Other Quest Schools 

a Our school has been directly involved in sharing successful practices with other Quest 
schools by either (1) adopting a practice that we learned about from a sister school 
(and)/or (2) helping a sister school adopt a practice that has been effectively used at our 

school. 

b. Individuals from our school have visited another Quest school or our school has hosted a 
visit by another Quest school to our campus. 

c Leadership team members have shared ideas with individuals from other schools at rallies 
and other network events. 

d Our school has had very limited involved with other Quest schools. 
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6. School Iniprovemcnt/Action Research 

a. Our school is implementing a school improvement project in connection with our 
involvement in Quest and is collecting data to demonstrate its impact on student learning. 

b. Our school is implementing a school improvement project in connection with our 
involvement in Quest, but has not designed a formal plan for assessing its effectiveness. 

c. Our school is currently considering one or more school improvement initiatives that 
would be supported by our involvement with Quest. 

d. Our school has no plans to pursue an improvement initiative as a part of our involvement 
in the Quest network. 



7. Change in School-Wide View of School Improvement 

a. Our school community has been significantly impacted by the Quest approach. A wide 
cross-section is aware of the Quest framework and committed to work on one or more of 
the component parts. 

b. Our faculty and staff have focused on one or more aspect of the Quest approach to 
continuous improvement. 

c. Members of the Quest leadership team have been affected by the Quest approach to 
continuous improvement. 

d. Our school community has not been influenced by the Quest approach to continuous 
improvement. 



8. Engagement in Related School Improvement Efforts 

a. Our school has been actively engaged in other school improvement efforts and the results 
have been recognizable. 

b. Our school has been involved in other school improvement efforts but there are no clearly 
identifiable results from our participation. 

c. Our school has not been involved in any other school improvement efforts. 



(c) Copyright, February 2000. AEL, Inc. Quest Project. 
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APPENDIX H: 



Completed Evaluation Standards Checklist 



Checklist for Applying the Standards 



To interpret the information provided on this form, the reader needs to refer to the full text of the standards as they appear in Joint 
Committee on Standards for Educational Evaluation, The Program Evaluation Standards (1994), Thousand Oaks, CA, Sage. 

The Standards were consulted and used as indicated in the table below (check as appropriate): 



Descriptor 

U1 Stakeholder Identification 


The Standard was 
addressed 

X 


The Standard was 
partially addressed 


The Standard was 
not addressed 


The Standard was 
not applicable 


U2 Evaluator Credibility 


X 








U3 Information Scope and Selection 


X 








U4 Values Identification 


X 








U5 Report Clarity 


X 








U6 Report Timeliness and Dissemination 


X 








U7 Evaluation Impact 


X 








FI Practical Procedures 


X 








F2 Political Viability 


X 








F3 Cost Effectiveness 


X 








PI Service Orientation 


X 








P2 Formal Agreements 


X 








P3 Rights of Human Subjects 


X 








P4 Human Interactions 


X 








P5 Complete and Fair Assessment 


X 








P6 Disclosure of Findings 


X 








P7 Conflict of Interest 


X 








P8 Fiscal Responsibility 


X 








A1 Program Documentation 


X 








A2 Context Analysis 


X 








A3 Described Purposes and Procedures 


X 








A4 Defensible Information Sources 


X 








A5 Valid Information 


X 








A6 Reliable Information 


X 








A7 Systematic Information 


X 








A8 Analysis of Quantitative Information 


X 








A9 Analysis of Qualitative Information 


X 








A10 Justified Conclusions 


X 








A 1 1 Impartial Reporting 


X 








A 12 Metaevaluation 


X 









The Program Evaluation Standards (1994, Sage) guided the development of this (check one): 

request for evaluation plan/design/proposal 

evaluation plan/design/proposal 

evaluation contract 

x evaluation report 

other: 

Name Caitlin Howley-Rowe Date 11/29/00 

^-O 

(signature) 

Position or Title Research Associate 

Agency AEL , Inc • * 

1031 Quarrier Street Charleston, WV 25301 

Relation to Document Author 

(e.g., author of document, evaluation team leader, external auditor, internal auditor) 
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U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
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